{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "dd7c4c87",
   "metadata": {},
   "source": [
    "# Lesson 1: Router Engine"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "877df988",
   "metadata": {},
   "source": [
    "Welcome to Lesson 1.\n",
    "\n",
    "To access the `requirements.txt` file, the data/pdf file required for this lesson and the `helper` and `utils` modules, please go to the `File` menu and select`Open...`.\n",
    "\n",
    "I hope you enjoy this course!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "357c97c9",
   "metadata": {},
   "source": [
    "## Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "ab5f4f4a-5890-451c-8869-24606ef9f396",
   "metadata": {
    "height": 63,
    "tags": []
   },
   "outputs": [],
   "source": [
    "from helper import get_openai_api_key\n",
    "\n",
    "OPENAI_API_KEY = get_openai_api_key()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "ffc9b4f4-64d4-4266-9889-54db90e00ee9",
   "metadata": {
    "height": 63,
    "tags": []
   },
   "outputs": [],
   "source": [
    "import nest_asyncio\n",
    "\n",
    "nest_asyncio.apply()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "49fca250",
   "metadata": {},
   "source": [
    "## Load Data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f3ae2a8c",
   "metadata": {
    "tags": []
   },
   "source": [
    "To download this paper, below is the needed code:\n",
    "\n",
    "#!wget \"https://openreview.net/pdf?id=VtmBAGCN7o\" -O metagpt.pdf\n",
    "\n",
    "**Note**: The pdf file is included with this lesson. To access it, go to the `File` menu and select`Open...`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "5c7f012d-dcd3-4881-a568-72dd27d79159",
   "metadata": {
    "height": 80,
    "tags": []
   },
   "outputs": [],
   "source": [
    "from llama_index.core import SimpleDirectoryReader\n",
    "\n",
    "# load documents\n",
    "documents = SimpleDirectoryReader(input_files=[\"metagpt.pdf\"]).load_data()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "f55865f9",
   "metadata": {
    "height": 29
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(29,\n",
       " Document(id_='f2f0eea8-eacc-40f7-8379-110187f03667', embedding=None, metadata={'page_label': '1', 'file_name': 'metagpt.pdf', 'file_path': 'metagpt.pdf', 'file_type': 'application/pdf', 'file_size': 16911937, 'creation_date': '2024-11-25', 'last_modified_date': '2024-06-24'}, excluded_embed_metadata_keys=['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], excluded_llm_metadata_keys=['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], relationships={}, text='Preprint\\nMETAGPT: M ETA PROGRAMMING FOR A\\nMULTI -AGENT COLLABORATIVE FRAMEWORK\\nSirui Hong1∗, Mingchen Zhuge2∗, Jonathan Chen1, Xiawu Zheng3, Yuheng Cheng4,\\nCeyao Zhang4,Jinlin Wang1,Zili Wang ,Steven Ka Shing Yau5,Zijuan Lin4,\\nLiyang Zhou6,Chenyu Ran1,Lingfeng Xiao1,7,Chenglin Wu1†,J¨urgen Schmidhuber2,8\\n1DeepWisdom,2AI Initiative, King Abdullah University of Science and Technology,\\n3Xiamen University,4The Chinese University of Hong Kong, Shenzhen,\\n5Nanjing University,6University of Pennsylvania,\\n7University of California, Berkeley,8The Swiss AI Lab IDSIA/USI/SUPSI\\nABSTRACT\\nRemarkable progress has been made on automated problem solving through so-\\ncieties of agents based on large language models (LLMs). Existing LLM-based\\nmulti-agent systems can already solve simple dialogue tasks. Solutions to more\\ncomplex tasks, however, are complicated through logic inconsistencies due to\\ncascading hallucinations caused by naively chaining LLMs. Here we introduce\\nMetaGPT, an innovative meta-programming framework incorporating efficient\\nhuman workflows into LLM-based multi-agent collaborations. MetaGPT en-\\ncodes Standardized Operating Procedures (SOPs) into prompt sequences for more\\nstreamlined workflows, thus allowing agents with human-like domain expertise\\nto verify intermediate results and reduce errors. MetaGPT utilizes an assembly\\nline paradigm to assign diverse roles to various agents, efficiently breaking down\\ncomplex tasks into subtasks involving many agents working together. On col-\\nlaborative software engineering benchmarks, MetaGPT generates more coherent\\nsolutions than previous chat-based multi-agent systems. Our project can be found\\nat https://github.com/geekan/MetaGPT.\\n1 I NTRODUCTION\\nAutonomous agents utilizing Large Language Models (LLMs) offer promising opportunities to en-\\nhance and replicate human workflows. In real-world applications, however, existing systems (Park\\net al., 2023; Zhuge et al., 2023; Cai et al., 2023; Wang et al., 2023c; Li et al., 2023; Du et al., 2023;\\nLiang et al., 2023; Hao et al., 2023) tend to oversimplify the complexities. They struggle to achieve\\neffective, coherent, and accurate problem-solving processes, particularly when there is a need for\\nmeaningful collaborative interaction (Chen et al., 2024; Zhang et al., 2023; Dong et al., 2023; Zhou\\net al., 2023; Qian et al., 2023).\\nThrough extensive collaborative practice, humans have developed widely accepted Standardized\\nOperating Procedures (SOPs) across various domains (Belbin, 2012; Manifesto, 2001; DeMarco &\\nLister, 2013). These SOPs play a critical role in supporting task decomposition and effective coor-\\ndination. Furthermore, SOPs outline the responsibilities of each team member, while establishing\\nstandards for intermediate outputs. Well-defined SOPs improve the consistent and accurate exe-\\ncution of tasks that align with defined roles and quality standards (Belbin, 2012; Manifesto, 2001;\\nDeMarco & Lister, 2013; Wooldridge & Jennings, 1998). For instance, in a software company,\\nProduct Managers analyze competition and user needs to create Product Requirements Documents\\n(PRDs) using a standardized structure, to guide the developmental process.\\nInspired by such ideas, we design a promising GPT -based Meta -Programming framework called\\nMetaGPT that significantly benefits from SOPs. Unlike other works (Li et al., 2023; Qian et al.,\\n2023), MetaGPT requires agents to generate structured outputs, such as high-quality requirements\\n∗These authors contributed equally to this work.\\n†Chenglin Wu (alexanderwu@fuzhi.ai) is the corresponding author, affiliated with DeepWisdom.\\n1', start_char_idx=None, end_char_idx=None, text_template='{metadata_str}\\n\\n{content}', metadata_template='{key}: {value}', metadata_seperator='\\n'))"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(documents), documents[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6b48a301",
   "metadata": {},
   "source": [
    "## Define LLM and Embedding model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "8a537bc0-78ee-4dda-a43f-60fd80062df6",
   "metadata": {
    "height": 80,
    "tags": []
   },
   "outputs": [],
   "source": [
    "from llama_index.core.node_parser import SentenceSplitter\n",
    "\n",
    "splitter = SentenceSplitter(chunk_size=1024)\n",
    "nodes = splitter.get_nodes_from_documents(documents)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "3a6e15e8",
   "metadata": {
    "height": 29
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[TextNode(id_='90141113-01d2-4fb0-b18a-89d2223948f7', embedding=None, metadata={'page_label': '1', 'file_name': 'metagpt.pdf', 'file_path': 'metagpt.pdf', 'file_type': 'application/pdf', 'file_size': 16911937, 'creation_date': '2024-11-25', 'last_modified_date': '2024-06-24'}, excluded_embed_metadata_keys=['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], excluded_llm_metadata_keys=['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], relationships={<NodeRelationship.SOURCE: '1'>: RelatedNodeInfo(node_id='f2f0eea8-eacc-40f7-8379-110187f03667', node_type=<ObjectType.DOCUMENT: '4'>, metadata={'page_label': '1', 'file_name': 'metagpt.pdf', 'file_path': 'metagpt.pdf', 'file_type': 'application/pdf', 'file_size': 16911937, 'creation_date': '2024-11-25', 'last_modified_date': '2024-06-24'}, hash='fbf95cf5a23181f54ac4dcdea1a2fc92e4f5b40c5c81190b0bbc69ac59a28b55'), <NodeRelationship.NEXT: '3'>: RelatedNodeInfo(node_id='37aefa64-74dd-4cfb-a65c-da3347a6adaf', node_type=<ObjectType.TEXT: '1'>, metadata={}, hash='f92bedddca4b2475904b4ae4c9d2ac788df4065083cb8ac5da0d26bfd014c48b')}, text='Preprint\\nMETAGPT: M ETA PROGRAMMING FOR A\\nMULTI -AGENT COLLABORATIVE FRAMEWORK\\nSirui Hong1∗, Mingchen Zhuge2∗, Jonathan Chen1, Xiawu Zheng3, Yuheng Cheng4,\\nCeyao Zhang4,Jinlin Wang1,Zili Wang ,Steven Ka Shing Yau5,Zijuan Lin4,\\nLiyang Zhou6,Chenyu Ran1,Lingfeng Xiao1,7,Chenglin Wu1†,J¨urgen Schmidhuber2,8\\n1DeepWisdom,2AI Initiative, King Abdullah University of Science and Technology,\\n3Xiamen University,4The Chinese University of Hong Kong, Shenzhen,\\n5Nanjing University,6University of Pennsylvania,\\n7University of California, Berkeley,8The Swiss AI Lab IDSIA/USI/SUPSI\\nABSTRACT\\nRemarkable progress has been made on automated problem solving through so-\\ncieties of agents based on large language models (LLMs). Existing LLM-based\\nmulti-agent systems can already solve simple dialogue tasks. Solutions to more\\ncomplex tasks, however, are complicated through logic inconsistencies due to\\ncascading hallucinations caused by naively chaining LLMs. Here we introduce\\nMetaGPT, an innovative meta-programming framework incorporating efficient\\nhuman workflows into LLM-based multi-agent collaborations. MetaGPT en-\\ncodes Standardized Operating Procedures (SOPs) into prompt sequences for more\\nstreamlined workflows, thus allowing agents with human-like domain expertise\\nto verify intermediate results and reduce errors. MetaGPT utilizes an assembly\\nline paradigm to assign diverse roles to various agents, efficiently breaking down\\ncomplex tasks into subtasks involving many agents working together. On col-\\nlaborative software engineering benchmarks, MetaGPT generates more coherent\\nsolutions than previous chat-based multi-agent systems. Our project can be found\\nat https://github.com/geekan/MetaGPT.\\n1 I NTRODUCTION\\nAutonomous agents utilizing Large Language Models (LLMs) offer promising opportunities to en-\\nhance and replicate human workflows. In real-world applications, however, existing systems (Park\\net al., 2023; Zhuge et al., 2023; Cai et al., 2023; Wang et al., 2023c; Li et al., 2023; Du et al., 2023;\\nLiang et al., 2023; Hao et al., 2023) tend to oversimplify the complexities. They struggle to achieve\\neffective, coherent, and accurate problem-solving processes, particularly when there is a need for\\nmeaningful collaborative interaction (Chen et al., 2024; Zhang et al., 2023; Dong et al., 2023; Zhou\\net al., 2023; Qian et al., 2023).\\nThrough extensive collaborative practice, humans have developed widely accepted Standardized\\nOperating Procedures (SOPs) across various domains (Belbin, 2012; Manifesto, 2001; DeMarco &\\nLister, 2013). These SOPs play a critical role in supporting task decomposition and effective coor-\\ndination. Furthermore, SOPs outline the responsibilities of each team member, while establishing\\nstandards for intermediate outputs. Well-defined SOPs improve the consistent and accurate exe-\\ncution of tasks that align with defined roles and quality standards (Belbin, 2012; Manifesto, 2001;\\nDeMarco & Lister, 2013; Wooldridge & Jennings, 1998). For instance, in a software company,\\nProduct Managers analyze competition and user needs to create Product Requirements Documents\\n(PRDs) using a standardized structure, to guide the developmental process.\\nInspired by such ideas, we design a promising GPT -based Meta -Programming framework called\\nMetaGPT that significantly benefits from SOPs. Unlike other works (Li et al., 2023; Qian et al.,\\n2023), MetaGPT requires agents to generate structured outputs, such as high-quality requirements\\n∗These authors contributed equally to this work.\\n†Chenglin Wu (alexanderwu@fuzhi.ai) is the corresponding author, affiliated with DeepWisdom.\\n1', start_char_idx=0, end_char_idx=3612, text_template='{metadata_str}\\n\\n{content}', metadata_template='{key}: {value}', metadata_seperator='\\n'),\n",
       " TextNode(id_='37aefa64-74dd-4cfb-a65c-da3347a6adaf', embedding=None, metadata={'page_label': '2', 'file_name': 'metagpt.pdf', 'file_path': 'metagpt.pdf', 'file_type': 'application/pdf', 'file_size': 16911937, 'creation_date': '2024-11-25', 'last_modified_date': '2024-06-24'}, excluded_embed_metadata_keys=['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], excluded_llm_metadata_keys=['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], relationships={<NodeRelationship.SOURCE: '1'>: RelatedNodeInfo(node_id='8984ce19-2aaa-4b5c-bb04-25547fb5d288', node_type=<ObjectType.DOCUMENT: '4'>, metadata={'page_label': '2', 'file_name': 'metagpt.pdf', 'file_path': 'metagpt.pdf', 'file_type': 'application/pdf', 'file_size': 16911937, 'creation_date': '2024-11-25', 'last_modified_date': '2024-06-24'}, hash='9fa2093ee11de8c10ce82694113f39f3e5bdf6e05ac8195b92dffb6902d926fa'), <NodeRelationship.PREVIOUS: '2'>: RelatedNodeInfo(node_id='90141113-01d2-4fb0-b18a-89d2223948f7', node_type=<ObjectType.TEXT: '1'>, metadata={'page_label': '1', 'file_name': 'metagpt.pdf', 'file_path': 'metagpt.pdf', 'file_type': 'application/pdf', 'file_size': 16911937, 'creation_date': '2024-11-25', 'last_modified_date': '2024-06-24'}, hash='fbf95cf5a23181f54ac4dcdea1a2fc92e4f5b40c5c81190b0bbc69ac59a28b55'), <NodeRelationship.NEXT: '3'>: RelatedNodeInfo(node_id='b4c40b5b-4c24-441f-8c7d-f47f70368501', node_type=<ObjectType.TEXT: '1'>, metadata={}, hash='7622cc60ba07cb07012cdc90d6a4eabd364da5fb76b2c5fd02e1931382e9258b')}, text='Preprint\\nFigure 1: The software development SOPs between MetaGPT and real-world human teams.\\nIn software engineering, SOPs promote collaboration among various roles. MetaGPT showcases\\nits ability to decompose complex tasks into specific actionable procedures assigned to various roles\\n(e.g., Product Manager, Architect, Engineer, etc.).\\ndocuments, design artifacts, flowcharts, and interface specifications. The use of intermediate struc-\\ntured outputs significantly increases the success rate of target code generation. Because it helps\\nmaintain consistency in communication, minimizing ambiguities and errors during collaboration.\\nMore graphically, in a company simulated by MetaGPT, all employees follow a strict and stream-\\nlined workflow, and all their handovers must comply with certain established standards. This reduces\\nthe risk of hallucinations caused by idle chatter between LLMs, particularly in role-playing frame-\\nworks, like: “ Hi, hello and how are you?” – Alice (Product Manager); “ Great! Have you had\\nlunch?” – Bob (Architect).\\nBenefiting from SOPs, MetaGPT offers a promising approach to meta-programming. In this context,\\nwe adopt meta-programming1as ”programming to program”, in contrast to the broader fields of meta\\nlearning and ”learning to learn” (Schmidhuber, 1987; 1993a; Hochreiter et al., 2001; Schmidhuber,\\n2006; Finn et al., 2017).\\nThis notion of meta-programming also encompasses earlier efforts like CodeBERT (Feng et al.,\\n2020) and recent projects such as CodeLlama (Rozi `ere et al., 2023) and WizardCoder (Luo\\net al., 2023). However, MetaGPT stands out as a unique solution that allows for efficient meta-\\nprogramming through a well-organized group of specialized agents. Each agent has a specific role\\nand expertise, following some established standards. This allows for automatic requirement analysis,\\nsystem design, code generation, modification, execution, and debugging during runtime, highlight-\\ning how agent-based techniques can enhance meta-programming.\\nTo validate the design of MetaGPT, we use publicly available HumanEval (Chen et al., 2021a) and\\nMBPP (Austin et al., 2021) for evaluations. Notably, in code generation benchmarks, MetaGPT\\nachieves a new state-of-the-art (SoTA) with 85.9% and 87.7% in Pass@1. When compared to other\\npopular frameworks for creating complex software projects, such as AutoGPT (Torantulino et al.,\\n2023), LangChain (Chase, 2022), AgentVerse (Chen et al., 2023), and ChatDev (Qian et al., 2023).\\nMetaGPT also stands out in handling higher levels of software complexity and offering extensive\\nfunctionality. Remarkably, in our experimental evaluations, MetaGPT achieves a 100% task com-\\npletion rate, demonstrating the robustness and efficiency (time and token costs) of our design.\\nWe summarize our contributions as follows:\\n1https://en.wikipedia.org/w/index.php?title=Metaprogramming\\n2', start_char_idx=0, end_char_idx=2866, text_template='{metadata_str}\\n\\n{content}', metadata_template='{key}: {value}', metadata_seperator='\\n'),\n",
       " TextNode(id_='b4c40b5b-4c24-441f-8c7d-f47f70368501', embedding=None, metadata={'page_label': '3', 'file_name': 'metagpt.pdf', 'file_path': 'metagpt.pdf', 'file_type': 'application/pdf', 'file_size': 16911937, 'creation_date': '2024-11-25', 'last_modified_date': '2024-06-24'}, excluded_embed_metadata_keys=['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], excluded_llm_metadata_keys=['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], relationships={<NodeRelationship.SOURCE: '1'>: RelatedNodeInfo(node_id='e58987db-9f50-46c6-8078-a3d2fa42f650', node_type=<ObjectType.DOCUMENT: '4'>, metadata={'page_label': '3', 'file_name': 'metagpt.pdf', 'file_path': 'metagpt.pdf', 'file_type': 'application/pdf', 'file_size': 16911937, 'creation_date': '2024-11-25', 'last_modified_date': '2024-06-24'}, hash='607830704f8e22038f172f4bf3faae8936e4d80ec66bd4ec1ff0406048a1c4a8'), <NodeRelationship.PREVIOUS: '2'>: RelatedNodeInfo(node_id='37aefa64-74dd-4cfb-a65c-da3347a6adaf', node_type=<ObjectType.TEXT: '1'>, metadata={'page_label': '2', 'file_name': 'metagpt.pdf', 'file_path': 'metagpt.pdf', 'file_type': 'application/pdf', 'file_size': 16911937, 'creation_date': '2024-11-25', 'last_modified_date': '2024-06-24'}, hash='9fa2093ee11de8c10ce82694113f39f3e5bdf6e05ac8195b92dffb6902d926fa'), <NodeRelationship.NEXT: '3'>: RelatedNodeInfo(node_id='dbe1c2e3-b561-484b-9261-95cab2d1590e', node_type=<ObjectType.TEXT: '1'>, metadata={}, hash='055dc5e4c33b187272421cfea1aa2f338b8fe0c864fabcd40ad7b4c4d73fc9fa')}, text='Preprint\\n•We introduce MetaGPT, a meta-programming framework for multi-agent collaboration based on\\nLLMs. It is highly convenient and flexible, with well-defined functions like role definition and\\nmessage sharing, making it a useful platform for developing LLM-based multi-agent systems.\\n•Our innovative integration of human-like SOPs throughout MetaGPT’s design significantly en-\\nhances its robustness, reducing unproductive collaboration among LLM-based agents. Furthermore,\\nwe introduce a novel executive feedback mechanism that debugs and executes code during runtime,\\nsignificantly elevating code generation quality (e.g., 5.4% absolute improvement on MBPP).\\n•We achieve state-of-the-art performance on HumanEval (Chen et al., 2021a) and MBPP (Austin\\net al., 2021). Extensive results convincingly validate MetaGPT, suggesting that it is a promising\\nmeta-programming framework for developing LLM-based multi-agent systems.\\n2 R ELATED WORK\\nAutomatic Programming The roots of automatic programming reach back deep into the previ-\\nous century. In 1969, Waldinger & Lee (1969) introduced “PROW,” a system designed to accept\\nprogram specifications written in predicate calculus, generate algorithms, and create LISP imple-\\nmentations (McCarthy, 1978). Balzer (1985) and Soloway (1986) made efforts to advance auto-\\nmatic programming and identified potential methods to achieve it. Recent approaches use natural\\nlanguage processing (NLP) techniques (Ni et al., 2023; Skreta et al., 2023; Feng et al., 2020; Li\\net al., 2022; Chen et al., 2018; 2021b; Zhang et al., 2023). Automatic programming has grown into\\nan industry delivering paid functions such as Microsoft Copilot. Lately, LLMs-based agents (Yao\\net al., 2022; Shinn et al., 2023; Lin et al., 2023) have advanced automatic programming develop-\\nment. Among them, ReAct (Yao et al., 2022) and Reflexion (Shinn et al., 2023) utilize a chain of\\nthought prompts (Wei et al., 2022) to generate reasoning trajectories and action plans with LLMs.\\nBoth works demonstrate the effectiveness of the ReAct style loop of reasoning as a design paradigm\\nfor empowering automatic programming. Additionally, ToolFormer (Schick et al., 2023) can learn\\nhow to use external tools through simple APIs. The research most closely aligned with our work\\nby Li et al. (2023) proposes a straightforward role-play framework for programming that involves\\ncommunication between agents playing different roles. Qian et al. (2023) utilizes multiple agents for\\nsoftware development. Although existing papers (Li et al., 2023; Qian et al., 2023) have improved\\nproductivity, they have not fully tapped into effective workflows with structured output formats.\\nThis makes it harder to deal with complex software engineering issues.\\nLLM-Based Multi-Agent Frameworks Recently, LLM-based autonomous agents have gained\\ntremendous interest in both industry and academia (Wang et al., 2023b). Many works (Chen et al.,\\n2024; Wang et al., 2023c; Du et al., 2023; Zhuge et al., 2023; Hao et al., 2023; Akata et al., 2023)\\nhave improved the problem-solving abilities of LLMs by integrating discussions among multiple\\nagents. Stable-Alignment (Liu et al., 2023) creates instruction datasets by deriving consensus on\\nvalue judgments through interactions across a sandbox with LLM agents. Other works focus on\\nsociological phenomena. For example, Generative Agents (Park et al., 2023) creates a “town” of 25\\nagents to study language interaction, social understanding, and collective memory. In the Natural\\nLanguage-Based Society of Mind (NLSOM) (Zhuge et al., 2023), agents with different functions\\ninteract to solve complex tasks through multiple rounds of “mindstorms.” Cai et al. (2023) propose\\na model for cost reduction by combining large models as tool makers and small models as tool users.\\nSome works emphasize cooperation and competition related to planning and strategy (Bakhtin et al.,\\n2022); others propose LLM-based economies (Zhuge et al., 2023). These works focus on open-\\nworld human behavior simulation, while MetaGPT aims to introduce human practice into multi-\\nagents frameworks.', start_char_idx=0, end_char_idx=4102, text_template='{metadata_str}\\n\\n{content}', metadata_template='{key}: {value}', metadata_seperator='\\n'),\n",
       " TextNode(id_='dbe1c2e3-b561-484b-9261-95cab2d1590e', embedding=None, metadata={'page_label': '3', 'file_name': 'metagpt.pdf', 'file_path': 'metagpt.pdf', 'file_type': 'application/pdf', 'file_size': 16911937, 'creation_date': '2024-11-25', 'last_modified_date': '2024-06-24'}, excluded_embed_metadata_keys=['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], excluded_llm_metadata_keys=['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], relationships={<NodeRelationship.SOURCE: '1'>: RelatedNodeInfo(node_id='e58987db-9f50-46c6-8078-a3d2fa42f650', node_type=<ObjectType.DOCUMENT: '4'>, metadata={'page_label': '3', 'file_name': 'metagpt.pdf', 'file_path': 'metagpt.pdf', 'file_type': 'application/pdf', 'file_size': 16911937, 'creation_date': '2024-11-25', 'last_modified_date': '2024-06-24'}, hash='607830704f8e22038f172f4bf3faae8936e4d80ec66bd4ec1ff0406048a1c4a8'), <NodeRelationship.PREVIOUS: '2'>: RelatedNodeInfo(node_id='b4c40b5b-4c24-441f-8c7d-f47f70368501', node_type=<ObjectType.TEXT: '1'>, metadata={'page_label': '3', 'file_name': 'metagpt.pdf', 'file_path': 'metagpt.pdf', 'file_type': 'application/pdf', 'file_size': 16911937, 'creation_date': '2024-11-25', 'last_modified_date': '2024-06-24'}, hash='4cff476a2394e254b437fbd8a75c1f84d8d14adb66e1eb8465fa5662c44407ec'), <NodeRelationship.NEXT: '3'>: RelatedNodeInfo(node_id='7cb8ea73-7aec-4cd3-ace5-e72fe304e4fa', node_type=<ObjectType.TEXT: '1'>, metadata={}, hash='822b9838f0d19770b636238e6b2e7c117feff5a7291cbe3dc0ff5cf77be55432')}, text='Other works focus on\\nsociological phenomena. For example, Generative Agents (Park et al., 2023) creates a “town” of 25\\nagents to study language interaction, social understanding, and collective memory. In the Natural\\nLanguage-Based Society of Mind (NLSOM) (Zhuge et al., 2023), agents with different functions\\ninteract to solve complex tasks through multiple rounds of “mindstorms.” Cai et al. (2023) propose\\na model for cost reduction by combining large models as tool makers and small models as tool users.\\nSome works emphasize cooperation and competition related to planning and strategy (Bakhtin et al.,\\n2022); others propose LLM-based economies (Zhuge et al., 2023). These works focus on open-\\nworld human behavior simulation, while MetaGPT aims to introduce human practice into multi-\\nagents frameworks. Besides, LLM-based agents face the challenges of “assistant repeated instruc-\\ntion” or “infinite loop of message” (Talebirad & Nadiri, 2023; Li et al., 2023). These challenges\\nbecome more urgent in task-oriented collaborations, which require consistent and mutually benefi-\\ncial interactions (Elazar et al., 2021; Wang et al., 2022; Jiang et al., 2023). This motivates our focus\\non applying advanced concepts such as Standard Operating Procedures in software development to\\nmulti-agent frameworks.\\n3', start_char_idx=3293, end_char_idx=4602, text_template='{metadata_str}\\n\\n{content}', metadata_template='{key}: {value}', metadata_seperator='\\n'),\n",
       " TextNode(id_='7cb8ea73-7aec-4cd3-ace5-e72fe304e4fa', embedding=None, metadata={'page_label': '4', 'file_name': 'metagpt.pdf', 'file_path': 'metagpt.pdf', 'file_type': 'application/pdf', 'file_size': 16911937, 'creation_date': '2024-11-25', 'last_modified_date': '2024-06-24'}, excluded_embed_metadata_keys=['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], excluded_llm_metadata_keys=['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], relationships={<NodeRelationship.SOURCE: '1'>: RelatedNodeInfo(node_id='7249089b-edd8-46f7-8ebd-bb4ca2d62235', node_type=<ObjectType.DOCUMENT: '4'>, metadata={'page_label': '4', 'file_name': 'metagpt.pdf', 'file_path': 'metagpt.pdf', 'file_type': 'application/pdf', 'file_size': 16911937, 'creation_date': '2024-11-25', 'last_modified_date': '2024-06-24'}, hash='d74b5ed1e3180bfc7ca6c65b9aabcced874283655cfe6aa01e05169df2e2f682'), <NodeRelationship.PREVIOUS: '2'>: RelatedNodeInfo(node_id='dbe1c2e3-b561-484b-9261-95cab2d1590e', node_type=<ObjectType.TEXT: '1'>, metadata={'page_label': '3', 'file_name': 'metagpt.pdf', 'file_path': 'metagpt.pdf', 'file_type': 'application/pdf', 'file_size': 16911937, 'creation_date': '2024-11-25', 'last_modified_date': '2024-06-24'}, hash='2a7f0566311e60053f70ccde9c695a072ea0063a038556036c63e4c4e90cbff5'), <NodeRelationship.NEXT: '3'>: RelatedNodeInfo(node_id='d3dd388e-7cfd-49bb-8201-b9e90db2b5a6', node_type=<ObjectType.TEXT: '1'>, metadata={}, hash='5c7ef438888f09961430c653251af7c5ec2771fb804b9e16586aac05a80b0a91')}, text='Preprint\\nFigure 2: An example of the communication protocol (left) and iterative programming with exe-\\ncutable feedback (right). Left: Agents use a shared message pool to publish structured messages.\\nThey can also subscribe to relevant messages based on their profiles. Right : After generating the\\ninitial code, the Engineer agent runs and checks for errors. If errors occur, the agent checks past\\nmessages stored in memory and compares them with the PRD, system design, and code files.\\n3 M ETAGPT: A M ETA-PROGRAMMING FRAMEWORK\\nMetaGPT is a meta-programming framework for LLM-based multi-agent systems. Sec. 3.1 pro-\\nvides an explanation of role specialization, workflow and structured communication in this frame-\\nwork, and illustrates how to organize a multi-agent system within the context of SOPs. Sec. 3.2\\npresents a communication protocol that enhances role communication efficiency. We also imple-\\nment structured communication interfaces and an effective publish-subscribe mechanism. These\\nmethods enable agents to obtain directional information from other roles and public information\\nfrom the environment. Finally, we introduce executable feedback—a self-correction mechanism for\\nfurther enhancing code generation quality during run-time in Sec. 3.3.\\n3.1 A GENTS IN STANDARD OPERATING PROCEDURES\\nSpecialization of Roles Unambiguous role specialization enables the breakdown of complex work\\ninto smaller and more specific tasks. Solving complex tasks or problems often requires the collab-\\noration of agents with diverse skills and expertise, each contributing specialized outputs tailored to\\nspecific issues.\\nIn a software company, a Product Manager typically conducts business-oriented analysis and derives\\ninsights, while a software engineer is responsible for programming. We define five roles in our\\nsoftware company: Product Manager, Architect, Project Manager, Engineer, and QA Engineer, as\\nshown in Figure 1. In MetaGPT, we specify the agent’s profile, which includes their name, profile,\\ngoal, and constraints for each role. We also initialize the specific context and skills for each role.\\nFor instance, a Product Manager can use web search tools, while an Engineer can execute code, as\\nshown in Figure 2. All agents adhere to the React-style behavior as described in Yao et al. (2022).\\nEvery agent monitors the environment ( i.e., the message pool in MetaGPT) to spot important ob-\\nservations ( e.g.,, messages from other agents). These messages can either directly trigger actions or\\nassist in finishing the job.\\nWorkflow across Agents By defining the agents’ roles and operational skills, we can establish\\nbasic workflows. In our work, we follow SOP in software development, which enables all agents to\\nwork in a sequential manner.\\n4', start_char_idx=0, end_char_idx=2757, text_template='{metadata_str}\\n\\n{content}', metadata_template='{key}: {value}', metadata_seperator='\\n'),\n",
       " TextNode(id_='d3dd388e-7cfd-49bb-8201-b9e90db2b5a6', embedding=None, metadata={'page_label': '5', 'file_name': 'metagpt.pdf', 'file_path': 'metagpt.pdf', 'file_type': 'application/pdf', 'file_size': 16911937, 'creation_date': '2024-11-25', 'last_modified_date': '2024-06-24'}, excluded_embed_metadata_keys=['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], excluded_llm_metadata_keys=['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], relationships={<NodeRelationship.SOURCE: '1'>: RelatedNodeInfo(node_id='83e65a1b-6bfd-4ba7-877b-4fcf3d7ef7cd', node_type=<ObjectType.DOCUMENT: '4'>, metadata={'page_label': '5', 'file_name': 'metagpt.pdf', 'file_path': 'metagpt.pdf', 'file_type': 'application/pdf', 'file_size': 16911937, 'creation_date': '2024-11-25', 'last_modified_date': '2024-06-24'}, hash='47acd3f043bc8c18b88ae2ffcca4cba124b154223bb4ac7ec709666d6169d171'), <NodeRelationship.PREVIOUS: '2'>: RelatedNodeInfo(node_id='7cb8ea73-7aec-4cd3-ace5-e72fe304e4fa', node_type=<ObjectType.TEXT: '1'>, metadata={'page_label': '4', 'file_name': 'metagpt.pdf', 'file_path': 'metagpt.pdf', 'file_type': 'application/pdf', 'file_size': 16911937, 'creation_date': '2024-11-25', 'last_modified_date': '2024-06-24'}, hash='d74b5ed1e3180bfc7ca6c65b9aabcced874283655cfe6aa01e05169df2e2f682'), <NodeRelationship.NEXT: '3'>: RelatedNodeInfo(node_id='df403f9e-1b1b-43e3-949e-9e031adddf1a', node_type=<ObjectType.TEXT: '1'>, metadata={}, hash='74ec265788faa78db040f07e9aa981ef95a1924af9bedb28693db754dacb67e9')}, text='Preprint\\nFigure 3: A diagram showing the software development process in MetaGPT, emphasizing its sig-\\nnificant dependence on SOPs. The more detailed demonstration can be found in Appendix B.\\nSpecifically, as shown in Figure 1, upon obtaining user requirements, the Product Manager under-\\ntakes a thorough analysis, formulating a detailed PRD that includes User Stories and Requirement\\nPool. This serves as a preliminary functional breakdown. The structured PRD is then passed to\\nthe Architect, who translates the requirements into system design components, such as File Lists,\\nData Structures, and Interface Definitions. Once captured in the system design, the information is\\ndirected towards the Project Manager for task distribution. Engineers proceed to execute the des-\\nignated classes and functions as outlined (detailed in Figure 2). In the following stage, the QA\\nEngineer formulates test cases to enforce stringent code quality. In the final step, MetaGPT pro-\\nduces a meticulously crafted software solution. We provide a detailed schematic (Figure 3) and a\\nconcrete instance (Appendix B) of the SOP workflow in MetaGPT.\\n3.2 C OMMUNICATION PROTOCOL\\nStructured Communication Interfaces Most current LLM-based multi-agent frameworks (Li\\net al., 2023; Zhuge et al., 2023; Zhang et al., 2023; Park et al., 2023) utilize unconstrained natural\\nlanguage as a communication interface.\\nHowever, despite the versatility of natural language, a question arises: does pure natural language\\ncommunication suffice for solving complex tasks? For example, in the telephone game (or Chinese\\n5', start_char_idx=0, end_char_idx=1583, text_template='{metadata_str}\\n\\n{content}', metadata_template='{key}: {value}', metadata_seperator='\\n'),\n",
       " TextNode(id_='df403f9e-1b1b-43e3-949e-9e031adddf1a', embedding=None, metadata={'page_label': '6', 'file_name': 'metagpt.pdf', 'file_path': 'metagpt.pdf', 'file_type': 'application/pdf', 'file_size': 16911937, 'creation_date': '2024-11-25', 'last_modified_date': '2024-06-24'}, excluded_embed_metadata_keys=['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], excluded_llm_metadata_keys=['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], relationships={<NodeRelationship.SOURCE: '1'>: RelatedNodeInfo(node_id='97c8fcda-99ad-4f33-a9bf-fc515e7170fc', node_type=<ObjectType.DOCUMENT: '4'>, metadata={'page_label': '6', 'file_name': 'metagpt.pdf', 'file_path': 'metagpt.pdf', 'file_type': 'application/pdf', 'file_size': 16911937, 'creation_date': '2024-11-25', 'last_modified_date': '2024-06-24'}, hash='f74ac928c7eced3e76759b74386fe6985b11ddd8d09b7b94e0332cea018cd732'), <NodeRelationship.PREVIOUS: '2'>: RelatedNodeInfo(node_id='d3dd388e-7cfd-49bb-8201-b9e90db2b5a6', node_type=<ObjectType.TEXT: '1'>, metadata={'page_label': '5', 'file_name': 'metagpt.pdf', 'file_path': 'metagpt.pdf', 'file_type': 'application/pdf', 'file_size': 16911937, 'creation_date': '2024-11-25', 'last_modified_date': '2024-06-24'}, hash='47acd3f043bc8c18b88ae2ffcca4cba124b154223bb4ac7ec709666d6169d171'), <NodeRelationship.NEXT: '3'>: RelatedNodeInfo(node_id='15ab87fc-3eef-438f-80f6-c6ad45df9eca', node_type=<ObjectType.TEXT: '1'>, metadata={}, hash='48613fd7eab64a34b5ecef742cdea90454ddf212e30dc0447ab36100754b2c5d')}, text='Preprint\\nwhispers)2, after several rounds of communication, the original information may be quite distorted.\\nInspired by human social structures, we propose using structured communication to formulate the\\ncommunication of agents. We establish a schema and format for each role and request that individ-\\nuals provide the necessary outputs based on their specific role and context.\\nAs shown in Figure 3, the Architect agent generates two outputs: the system interface design and a\\nsequence flow diagram. These contain system module design and interaction sequences, which serve\\nas important deliverables for Engineers. Unlike ChatDev (Zhao et al., 2023), agents in MetaGPT\\ncommunicate through documents and diagrams (structured outputs) rather than dialogue. These\\ndocuments contain all necessary information, preventing irrelevant or missing content.\\nPublish-Subscribe Mechanism Sharing information is critical in collaboration. For instance,\\nArchitects and Engineers often need to reference PRDs. However, communicating this information\\neach time in a one-to-one manner, as indicated by previous work (Li et al., 2023; Zhao et al., 2023;\\nZhang et al., 2023), can complicate the communication topology, resulting in inefficiencies.\\nTo address this challenge, a viable approach is to store information in a global message pool . As\\nshown in Figure 2 (left), we introduce a shared message pool that allows all agents to exchange\\nmessages directly. These agents not only publish their structured messages in the pool but also access\\nmessages from other entities transparently. Any agent can directly retrieve required information\\nfrom the shared pool, eliminating the need to inquire about other agents and await their responses.\\nThis enhances communication efficiency.\\nSharing all information with every agent can lead to information overload. During task execution,\\nan agent typically prefers to receive only task-related information and avoid distractions through\\nirrelevant details. Effective management and dissemination of this information play a crucial role.\\nWe offer a simple and effective solution- subscription mechanism (in Figure 2 (left)). Instead of\\nrelying on dialogue, agents utilize role-specific interests to extract relevant information. They can\\nselect information to follow based on their role profiles. In practical implementations, an agent\\nactivates its action only after receiving all its prerequisite dependencies. As illustrated in Figure 3,\\nthe Architect mainly focuses on PRDs provided by the Product Manager, while documents from\\nroles such as the QA Engineer might be of lesser concern.\\n3.3 I TERATIVE PROGRAMMING WITH EXECUTABLE FEEDBACK\\nIn daily programming tasks, the processes of debugging and optimization play important roles.\\nHowever, existing methods often lack a self-correction mechanism, which leads to unsuccessful code\\ngeneration. Previous work introduced non-executable code review and self-reflection (Zhao et al.,\\n2023; Yao et al., 2022; Shinn et al., 2023; Dong et al., 2023). However, they still face challenges in\\nensuring code executability and runtime correctness.\\nOur first MetaGPT implementations overlooked certain errors during the review process, due to\\nLLM hallucinations (Manakul et al., 2023). To overcome this, after initial code generation, we\\nintroduce an executable feedback mechanism to improve the code iteratively. More specifically, as\\nshown in Figure 2, the Engineer is asked to write code based on the original product requirements\\nand design.\\nThis enables the Engineer to continuously improve code using its own historical execution and\\ndebugging memory. To obtain additional information, the Engineer writes and executes the corre-\\nsponding unit test cases, and subsequently receives the test results. If satisfactory, additional devel-\\nopment tasks are initiated. Otherwise the Engineer debugs the code before resuming programming.\\nThis iterative testing process continues until the test is passed or a maximum of 3 retries is reached.\\n4 E XPERIMENTS\\n4.1 E XPERIMENTAL SETTING\\nDatasets We use two public benchmarks, HumanEval (Chen et al., 2021a) and MBPP (Austin\\net al., 2021), and a self-generated, more challenging software development benchmark named Soft-\\n2https://en.wikipedia.org/wiki/Chinese whispers\\n6', start_char_idx=0, end_char_idx=4280, text_template='{metadata_str}\\n\\n{content}', metadata_template='{key}: {value}', metadata_seperator='\\n'),\n",
       " TextNode(id_='15ab87fc-3eef-438f-80f6-c6ad45df9eca', embedding=None, metadata={'page_label': '7', 'file_name': 'metagpt.pdf', 'file_path': 'metagpt.pdf', 'file_type': 'application/pdf', 'file_size': 16911937, 'creation_date': '2024-11-25', 'last_modified_date': '2024-06-24'}, excluded_embed_metadata_keys=['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], excluded_llm_metadata_keys=['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], relationships={<NodeRelationship.SOURCE: '1'>: RelatedNodeInfo(node_id='33b40042-507a-43d5-916d-645dd55941c0', node_type=<ObjectType.DOCUMENT: '4'>, metadata={'page_label': '7', 'file_name': 'metagpt.pdf', 'file_path': 'metagpt.pdf', 'file_type': 'application/pdf', 'file_size': 16911937, 'creation_date': '2024-11-25', 'last_modified_date': '2024-06-24'}, hash='1b56713386b9b77fa56e12576440083742126ef2490e921c97e4a0bb8641b183'), <NodeRelationship.PREVIOUS: '2'>: RelatedNodeInfo(node_id='df403f9e-1b1b-43e3-949e-9e031adddf1a', node_type=<ObjectType.TEXT: '1'>, metadata={'page_label': '6', 'file_name': 'metagpt.pdf', 'file_path': 'metagpt.pdf', 'file_type': 'application/pdf', 'file_size': 16911937, 'creation_date': '2024-11-25', 'last_modified_date': '2024-06-24'}, hash='f74ac928c7eced3e76759b74386fe6985b11ddd8d09b7b94e0332cea018cd732'), <NodeRelationship.NEXT: '3'>: RelatedNodeInfo(node_id='00847fb1-f77e-4b6c-ad5d-e1aff6b8d006', node_type=<ObjectType.TEXT: '1'>, metadata={}, hash='76006fa3a24f0109d89ab1945fd625fd4a5162026850e6268fb61645e8afa2f4')}, text='Preprint\\nwareDev: (1) HumanEval includes 164 handwritten programming tasks. These tasks encompass\\nfunction specifications, descriptions, reference codes, and tests. (2) MBPP consists of 427 Python\\ntasks. These tasks cover core concepts and standard library features and include descriptions, ref-\\nerence codes, and automated tests. (3) Our SoftwareDev dataset is a collection of 70 representa-\\ntive examples of software development tasks, each with its own task prompt (see Table 8). These\\ntasks have diverse scopes (See Figure 5), such as mini-games, image processing algorithms, data\\nvisualization. They offer a robust testbed for authentic development tasks. Contrary to previous\\ndatasets (Chen et al., 2021a; Austin et al., 2021), SoftwareDev focuses on the engineering aspects.\\nIn the comparisons, we randomly select seven representative tasks for evaluation.\\nEvaluation Metrics For HuamnEval and MBPP, we follow the unbiased version of Pass @ kas\\npresented by (Chen et al., 2021a; Dong et al., 2023), to evaluate the functional accuracy of the top-k\\ngenerated codes: Pass @ k=EProblems\\x14\\n1−(n−c\\nk)\\n(n\\nk)\\x15\\n.\\nFor SoftwareDev, we prioritize practical use and evaluate performance through human evaluations\\n(A, E) or statistical analysis (B, C, D): (A)Executability: this metric rates code from 1 (failure/non-\\nfunctional) to 4 (flawless). ‘1’ is for non-functional, ‘2’ for runnable but imperfect, ‘3’ for nearly\\nperfect, and ‘4’ for flawless code. (B)Cost: the cost evaluations here include the (1) running time,\\n(2) token usage, and (3) expenses. (C)Code Statistics: this includes (1) code files, (2) lines of code\\nper file, and (3) total code lines. (D)Productivity: basically, it is defined as the number of token\\nusage divided by the number of lines of code, which refers to the consumption of tokens per code\\nline. (E)Human Revision Cost: refers to times of manual code corrections, which tackle problems\\nlike package import errors, incorrect class names, or incomplete reference paths. Typically, each\\ncorrection involves up to 3 lines of code.\\nBaselines We compare our method with recent domain-specific LLMs in the code generation field,\\nincluding AlphaCode (Li et al., 2022), Incoder (Fried et al., 2022), CodeGeeX (Zheng et al., 2023),\\nCodeGen (Nijkamp et al., 2023), CodeX (Chen et al., 2021a), and CodeT (Chen et al., 2022) and\\ngeneral domain LLMs such as PaLM (Chowdhery et al., 2022), and GPT-4 (OpenAI, 2023). Several\\nresults of baselines (such as Incoder, CodeGeeX) are provided by Dong et al. (2023). In HumanEval\\nand MBPP, we slightly modified the prompts to align with response format requirements. These\\nmodifications aim to address format-specific issues (i.e., Python problems). With the SoftwareDev\\nbenchmark, we provide a comprehensive comparison between MetaGPT, AutoGPT (Torantulino\\net al., 2023), LangChain (Chase, 2022) with Python Read-Eval-Print Loop (REPL) tool3, Agent-\\nVerse (Chen et al., 2023), and ChatDev (Qian et al., 2023).\\n4.2 M AINRESULT\\nAlphaCode(1.1B)\\nIncoder (6.7B)\\nCodeGeeX (13B)17.1\\n—15.2 17.6 18.926.9\\nCodeGeeX-Mono(16.1B)32.938.6\\nGPT-467.0\\n—\\nMetaGPT\\n(w/o Feedback)81.7 82.3Pass@1 of MBPP  and HumanEval (%)\\nPaLM Coder(540B)36.047.0\\nCodex (175B)47.058.1\\nCodex + CodeT65.8 67.7\\nHumanEval\\nMBPP\\nMetaGPT85.9 87.7\\nFigure 4: Pass rates on the MBPP and HumanEval with a single attempt.\\nPerformance Figure 4 demonstrates that MetaGPT outperforms all preceding approaches in both\\nHumanEval and MBPP benchmarks.', start_char_idx=0, end_char_idx=3450, text_template='{metadata_str}\\n\\n{content}', metadata_template='{key}: {value}', metadata_seperator='\\n'),\n",
       " TextNode(id_='00847fb1-f77e-4b6c-ad5d-e1aff6b8d006', embedding=None, metadata={'page_label': '7', 'file_name': 'metagpt.pdf', 'file_path': 'metagpt.pdf', 'file_type': 'application/pdf', 'file_size': 16911937, 'creation_date': '2024-11-25', 'last_modified_date': '2024-06-24'}, excluded_embed_metadata_keys=['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], excluded_llm_metadata_keys=['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], relationships={<NodeRelationship.SOURCE: '1'>: RelatedNodeInfo(node_id='33b40042-507a-43d5-916d-645dd55941c0', node_type=<ObjectType.DOCUMENT: '4'>, metadata={'page_label': '7', 'file_name': 'metagpt.pdf', 'file_path': 'metagpt.pdf', 'file_type': 'application/pdf', 'file_size': 16911937, 'creation_date': '2024-11-25', 'last_modified_date': '2024-06-24'}, hash='1b56713386b9b77fa56e12576440083742126ef2490e921c97e4a0bb8641b183'), <NodeRelationship.PREVIOUS: '2'>: RelatedNodeInfo(node_id='15ab87fc-3eef-438f-80f6-c6ad45df9eca', node_type=<ObjectType.TEXT: '1'>, metadata={'page_label': '7', 'file_name': 'metagpt.pdf', 'file_path': 'metagpt.pdf', 'file_type': 'application/pdf', 'file_size': 16911937, 'creation_date': '2024-11-25', 'last_modified_date': '2024-06-24'}, hash='c9318da8eb4c4c741cf8f37b539a2ae4391b29661c6e2a20d22fd1468422b54a'), <NodeRelationship.NEXT: '3'>: RelatedNodeInfo(node_id='3d42ff6a-ba0b-4a52-a6d2-3a9989c5738c', node_type=<ObjectType.TEXT: '1'>, metadata={}, hash='026ae0c03a6d4c62c7b524d2004a1e36eda657e02b613f5d22a88aa5f941efbf')}, text='Performance Figure 4 demonstrates that MetaGPT outperforms all preceding approaches in both\\nHumanEval and MBPP benchmarks. When MetaGPT collaborates with GPT-4, it significantly im-\\nproves the Pass @ kin the HumanEval benchmark compared to GPT-4. It achieves 85.9% and 87.7%\\n3https://en.wikipedia.org/wiki/Read–eval–print loop\\n7', start_char_idx=3328, end_char_idx=3656, text_template='{metadata_str}\\n\\n{content}', metadata_template='{key}: {value}', metadata_seperator='\\n'),\n",
       " TextNode(id_='3d42ff6a-ba0b-4a52-a6d2-3a9989c5738c', embedding=None, metadata={'page_label': '8', 'file_name': 'metagpt.pdf', 'file_path': 'metagpt.pdf', 'file_type': 'application/pdf', 'file_size': 16911937, 'creation_date': '2024-11-25', 'last_modified_date': '2024-06-24'}, excluded_embed_metadata_keys=['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], excluded_llm_metadata_keys=['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], relationships={<NodeRelationship.SOURCE: '1'>: RelatedNodeInfo(node_id='0106d5b5-8f49-4ee9-86ba-677aadef04cd', node_type=<ObjectType.DOCUMENT: '4'>, metadata={'page_label': '8', 'file_name': 'metagpt.pdf', 'file_path': 'metagpt.pdf', 'file_type': 'application/pdf', 'file_size': 16911937, 'creation_date': '2024-11-25', 'last_modified_date': '2024-06-24'}, hash='041efafb4f20b72c4a8fd2191541db3e13a6ef2e4153e7f300d92e148fe665a4'), <NodeRelationship.PREVIOUS: '2'>: RelatedNodeInfo(node_id='00847fb1-f77e-4b6c-ad5d-e1aff6b8d006', node_type=<ObjectType.TEXT: '1'>, metadata={'page_label': '7', 'file_name': 'metagpt.pdf', 'file_path': 'metagpt.pdf', 'file_type': 'application/pdf', 'file_size': 16911937, 'creation_date': '2024-11-25', 'last_modified_date': '2024-06-24'}, hash='53e17bb1c7a723019f40ddef5f6dbb01f1a97dd0ab2c3aabb0cc15151b8bbbd6'), <NodeRelationship.NEXT: '3'>: RelatedNodeInfo(node_id='d999eaa8-8fd3-4ee9-8d43-89edc9f60a15', node_type=<ObjectType.TEXT: '1'>, metadata={}, hash='ba27173bb8ae82285144e3d71e035b4798731c4cf39f775dacbf466bd72f3ff5')}, text='Preprint\\nFigure 5: Demo softwares developed by MetaGPT.\\nin these two public benchmarks. Moreover, as shown in Table 1, MetaGPT outperforms ChatDev on\\nthe challenging SoftwareDev dataset in nearly all metrics. For example, considering the executabil-\\nity, MetaGPT achieves a score of 3.75, which is very close to 4 (flawless). Besides, it takes less time\\n(503 seconds), clearly less than ChatDev. Considering the code statistic and the cost of human revi-\\nsion, it also significantly outperforms ChatDev. Although MetaGPT requires more tokens (24,613\\nor 31,255 compared to 19,292), it needs only 126.5/124.3 tokens to generate one line of code. In\\ncontrast, ChatDev uses 248.9 tokens. These results highlight the benefits of SOPs in collabora-\\ntions between multiple agents. Additionally, we demonstrate the autonomous software generation\\ncapabilities of MetaGPT through visualization samples (Figure 5). For additional experiments and\\nanalysis, please refer to Appendix C.\\nTable 1: The statistical analysis on SoftwareDev.\\nStatistical Index ChatDev MetaGPT w/o Feedback MetaGPT\\n(A)Executability 2.25 3.67 3.75\\n(B)Cost#1: Running Times (s) 762 503 541\\n(B)Cost#2: Token Usage 19,292 24,613 31,255\\n(C)Code Statistic#1: Code Files 1.9 4.6 5.1\\n(C)Code Statistic#2: Lines of Code per File 40.8 42.3 49.3\\n(C)Code Statistic#3: Total Code Lines 77.5 194.6 251.4\\n(D)Productivity 248.9 126.5 124.3\\n(E)Human Revision Cost 2.5 2.25 0.83\\n4.3 C APABILITIES ANALYSIS\\nCompared to open-source baseline methods such as AutoGPT and autonomous agents such as\\nAgentVerse and ChatDev, MetaGPT offers functions for software engineering tasks. As presented\\nin Table 2, our framework encompasses a wide range of abilities to handle complex and specialized\\ndevelopment tasks efficiently. Incorporating SOPs (e.g., role-play expertise, structured communi-\\ncation, streamlined workflow) can significantly improve code generation. Other baseline methods\\n8', start_char_idx=0, end_char_idx=1925, text_template='{metadata_str}\\n\\n{content}', metadata_template='{key}: {value}', metadata_seperator='\\n'),\n",
       " TextNode(id_='d999eaa8-8fd3-4ee9-8d43-89edc9f60a15', embedding=None, metadata={'page_label': '9', 'file_name': 'metagpt.pdf', 'file_path': 'metagpt.pdf', 'file_type': 'application/pdf', 'file_size': 16911937, 'creation_date': '2024-11-25', 'last_modified_date': '2024-06-24'}, excluded_embed_metadata_keys=['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], excluded_llm_metadata_keys=['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], relationships={<NodeRelationship.SOURCE: '1'>: RelatedNodeInfo(node_id='44d64ea4-fcd2-4d2e-b58b-21e43c5a27dc', node_type=<ObjectType.DOCUMENT: '4'>, metadata={'page_label': '9', 'file_name': 'metagpt.pdf', 'file_path': 'metagpt.pdf', 'file_type': 'application/pdf', 'file_size': 16911937, 'creation_date': '2024-11-25', 'last_modified_date': '2024-06-24'}, hash='3e5f92fda7c6fe6f86ffe0362adfa7f46d96800b41c4945a57eb718618da6cb0'), <NodeRelationship.PREVIOUS: '2'>: RelatedNodeInfo(node_id='3d42ff6a-ba0b-4a52-a6d2-3a9989c5738c', node_type=<ObjectType.TEXT: '1'>, metadata={'page_label': '8', 'file_name': 'metagpt.pdf', 'file_path': 'metagpt.pdf', 'file_type': 'application/pdf', 'file_size': 16911937, 'creation_date': '2024-11-25', 'last_modified_date': '2024-06-24'}, hash='041efafb4f20b72c4a8fd2191541db3e13a6ef2e4153e7f300d92e148fe665a4'), <NodeRelationship.NEXT: '3'>: RelatedNodeInfo(node_id='dd662376-11bc-4281-81d7-fb7c32cd038f', node_type=<ObjectType.TEXT: '1'>, metadata={}, hash='b2f651c1dbef992e6849b24db89e890a149b43c33e6476fb8460cbb58bf44840')}, text='Preprint\\nTable 2: Comparison of capabilities for MetaGPT and other approaches. ‘!’ indicates the\\npresence of a specific feature in the corresponding framework, ‘ %’ its absence.\\nFramework Capabiliy AutoGPT LangChain AgentVerse ChatDev MetaGPT\\nPRD generation % % % % !\\nTenical design genenration % % % % !\\nAPI interface generation % % % % !\\nCode generation ! ! ! ! !\\nPrecompilation execution % % % % !\\nRole-based task management % % % ! !\\nCode review % % ! ! !\\nTable 3: Ablation study on roles. ‘#’ denotes ‘The number of’, ‘Product’ denotes ‘Product man-\\nager’, and ‘Project’ denotes ‘Project manager’. ‘ !’ indicates the addition of a specific role. ‘Revi-\\nsions’ refers to ‘Human Revision Cost’.\\nEngineer Product Architect Project #Agents #Lines Expense Revisions Executability\\n! % % % 1 83.0 $ 0.915 10 1.0\\n! ! % % 2 112.0 $ 1.059 6.5 2.0\\n! ! ! % 3 143.0 $ 1.204 4.0 2.5\\n! ! % ! 3 205.0 $ 1.251 3.5 2.0\\n! ! ! ! 4 191.0 $ 1.385 2.5 4.0\\ncan easily integrate SOP-like designs to improve their performance, similar to injecting chain-of-\\nthought (Wei et al., 2022) in LLMs.\\n4.4 A BLATION STUDY\\nThe Effectiveness of Roles To understand the impact of different roles on the final results, we\\nperform two tasks that involve generating effective code and calculating average statistics. When we\\nexclude certain roles, unworkable codes are generated. As indicated by Table 3, the addition of roles\\ndifferent from just the Engineer consistently improves both revisions and executability. While more\\nroles slightly increase the expenses, the overall performance improves noticeably, demonstrating the\\neffectiveness of the various roles.\\nThe Effectiveness of Executable Feedback Mechanism As shown in Figure 4, adding executable\\nfeedback into MetaGPT leads to a significant improvement of 4.2% and 5.4% in Pass @1 on Hu-\\nmanEval and MBPP, respectively. Besides, Table 1 shows that the feedback mechanism improves\\nfeasibility (3.67 to 3.75) and reduces the cost of human revisions (2.25 to 0.83). These results\\nillustrate how our designed feedback mechanism can produce higher-quality code. Additional quan-\\ntitative results of MetaGPT and MetaGPT without executable feedback are shown in Table 4 and\\nTable 9.\\n5 C ONCLUSION\\nThis work introduces MetaGPT, a novel meta-programming framework that leverages SOPs to en-\\nhance the problem-solving capabilities of multi-agent systems based on Large Language Models\\n(LLMs). MetaGPT models a group of agents as a simulated software company, analogous to simu-\\nlated towns (Park et al., 2023) and the Minecraft Sandbox in V oyager (Wang et al., 2023a). MetaGPT\\nleverages role specialization, workflow management, and efficient sharing mechanisms such as mes-\\nsage pools and subscriptions, rendering it a flexible and portable platform for autonomous agents\\nand multi-agent frameworks. It uses an executable feedback mechanism to enhance code generation\\nquality during runtime. In extensive experiments, MetaGPT achieves state-of-the-art performance\\non multiple benchmarks. The successful integration of human-like SOPs inspires future research\\non human-inspired techniques for artificial multi-agent systems. We also view our work as an early\\nattempt to regulate LLM-based multi-agent frameworks. See also the outlook (Appendix A) .\\n9', start_char_idx=0, end_char_idx=3265, text_template='{metadata_str}\\n\\n{content}', metadata_template='{key}: {value}', metadata_seperator='\\n'),\n",
       " TextNode(id_='dd662376-11bc-4281-81d7-fb7c32cd038f', embedding=None, metadata={'page_label': '10', 'file_name': 'metagpt.pdf', 'file_path': 'metagpt.pdf', 'file_type': 'application/pdf', 'file_size': 16911937, 'creation_date': '2024-11-25', 'last_modified_date': '2024-06-24'}, excluded_embed_metadata_keys=['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], excluded_llm_metadata_keys=['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], relationships={<NodeRelationship.SOURCE: '1'>: RelatedNodeInfo(node_id='01b4697f-e524-4b85-9300-05e73ae52617', node_type=<ObjectType.DOCUMENT: '4'>, metadata={'page_label': '10', 'file_name': 'metagpt.pdf', 'file_path': 'metagpt.pdf', 'file_type': 'application/pdf', 'file_size': 16911937, 'creation_date': '2024-11-25', 'last_modified_date': '2024-06-24'}, hash='7d60a07cf9f298aaa6c5b3d40a4633848162fd32c58407c7226fedba03c0596b'), <NodeRelationship.PREVIOUS: '2'>: RelatedNodeInfo(node_id='d999eaa8-8fd3-4ee9-8d43-89edc9f60a15', node_type=<ObjectType.TEXT: '1'>, metadata={'page_label': '9', 'file_name': 'metagpt.pdf', 'file_path': 'metagpt.pdf', 'file_type': 'application/pdf', 'file_size': 16911937, 'creation_date': '2024-11-25', 'last_modified_date': '2024-06-24'}, hash='3e5f92fda7c6fe6f86ffe0362adfa7f46d96800b41c4945a57eb718618da6cb0'), <NodeRelationship.NEXT: '3'>: RelatedNodeInfo(node_id='e1eebe71-b88c-436c-bf9c-fe1d1b53db2c', node_type=<ObjectType.TEXT: '1'>, metadata={}, hash='c3285b8059c5f308c8ef62ab51fc003b1ac0d1dceead1bb36bc5cb56210f91da')}, text='Preprint\\nAcknowledgement\\nWe thank Sarah Salhi, the Executive Secretary of KAUST AI Initiative, and Yuhui Wang, Postdoc-\\ntoral Fellow at the KAUST AI Initiative, for helping to polish some of the text. We would like to\\nexpress our gratitude to Wenyi Wang, a PhD student at the KAUST AI Initiative, for providing com-\\nprehensive feedback on the paper and for helping to draft the outlook (Appendix A) with Mingchen.\\nWe also thank Zongze Xu, the vice president of DeepWisdom, for providing illustrative materials\\nfor AgentStore.\\nAuthor Contributions\\nSirui Hong conducted most of the experiments and designed the executable feedback module. She\\nalso led the initial version of the write-up, supported by Ceyao Zhang, and also by Jinlin Wang and\\nZili Wang. Mingchen Zhuge designed the self-improvement module, discussed additional experi-\\nments, and led the current write-up. Jonathan Chen helped with the MBPP experiments, outlined\\nthe methods section, and contributed to the current write-up. Xiawu Zheng provided valuable guid-\\nance, reviewed and edited the paper. Yuheng Cheng contributed to the evaluation metric design and\\nHumanEval experiments. Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, Lingfeng Xiao helped\\nwith the MBPP experiments and comparisons to open-source baseline methods. Chenyu Ran cre-\\nated most of the illustrative figures. Chenglin Wu is the CEO of DeepWisdom, initiated MetaGPT,\\nmade the most significant code contributions to it, and advised this project. J ¨urgen Schmidhuber,\\nDirector of the AI Initiative at KAUST and Scientific Director of IDSIA, advised this project and\\nhelped with the write-up.\\nREFERENCES\\nElif Akata, Lion Schulz, Julian Coda-Forno, Seong Joon Oh, Matthias Bethge, and Eric Schulz.\\nPlaying repeated games with large language models. arXiv preprint , 2023.\\nJacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan,\\nEllen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. Program synthesis with large\\nlanguage models, 2021.\\nAnton Bakhtin, Noam Brown, Emily Dinan, Gabriele Farina, Colin Flaherty, Daniel Fried, Andrew\\nGoff, Jonathan Gray, Hengyuan Hu, et al. Human-level play in the game of diplomacy by com-\\nbining language models with strategic reasoning. Science , 2022.\\nRobert Balzer. A 15 year perspective on automatic programming. TSE, 1985.\\nR.M. Belbin. Team Roles at Work . Routledge, 2012. URL https://books.google.co.uk/\\nbooks?id=MHIQBAAAQBAJ .\\nTianle Cai, Xuezhi Wang, Tengyu Ma, Xinyun Chen, and Denny Zhou. Large language models as\\ntool makers. arXiv preprint , 2023.\\nHarrison Chase. LangChain. https://github.com/hwchase17/langchain , 2022.\\nBei Chen, Fengji Zhang, Anh Nguyen, Daoguang Zan, Zeqi Lin, Jian-Guang Lou, and Weizhu\\nChen. Codet: Code generation with generated tests, 2022.\\nJiaqi Chen, Yuxian Jiang, Jiachen Lu, and Li Zhang. S-agents: self-organizing agents in open-ended\\nenvironment. arXiv preprint , 2024.', start_char_idx=0, end_char_idx=2924, text_template='{metadata_str}\\n\\n{content}', metadata_template='{key}: {value}', metadata_seperator='\\n'),\n",
       " TextNode(id_='e1eebe71-b88c-436c-bf9c-fe1d1b53db2c', embedding=None, metadata={'page_label': '10', 'file_name': 'metagpt.pdf', 'file_path': 'metagpt.pdf', 'file_type': 'application/pdf', 'file_size': 16911937, 'creation_date': '2024-11-25', 'last_modified_date': '2024-06-24'}, excluded_embed_metadata_keys=['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], excluded_llm_metadata_keys=['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], relationships={<NodeRelationship.SOURCE: '1'>: RelatedNodeInfo(node_id='01b4697f-e524-4b85-9300-05e73ae52617', node_type=<ObjectType.DOCUMENT: '4'>, metadata={'page_label': '10', 'file_name': 'metagpt.pdf', 'file_path': 'metagpt.pdf', 'file_type': 'application/pdf', 'file_size': 16911937, 'creation_date': '2024-11-25', 'last_modified_date': '2024-06-24'}, hash='7d60a07cf9f298aaa6c5b3d40a4633848162fd32c58407c7226fedba03c0596b'), <NodeRelationship.PREVIOUS: '2'>: RelatedNodeInfo(node_id='dd662376-11bc-4281-81d7-fb7c32cd038f', node_type=<ObjectType.TEXT: '1'>, metadata={'page_label': '10', 'file_name': 'metagpt.pdf', 'file_path': 'metagpt.pdf', 'file_type': 'application/pdf', 'file_size': 16911937, 'creation_date': '2024-11-25', 'last_modified_date': '2024-06-24'}, hash='aa2ed072af4602d40b01b8c3cdf69f807e17ebde78b0a4af5c955699eedeeb3c'), <NodeRelationship.NEXT: '3'>: RelatedNodeInfo(node_id='32ce48bd-a65c-4acd-87ac-f2d96acd2a64', node_type=<ObjectType.TEXT: '1'>, metadata={}, hash='42237a63dfc11b3486c316b6a4b5c6f04f0f584542958aa367ec2fc26107e771')}, text='Team Roles at Work . Routledge, 2012. URL https://books.google.co.uk/\\nbooks?id=MHIQBAAAQBAJ .\\nTianle Cai, Xuezhi Wang, Tengyu Ma, Xinyun Chen, and Denny Zhou. Large language models as\\ntool makers. arXiv preprint , 2023.\\nHarrison Chase. LangChain. https://github.com/hwchase17/langchain , 2022.\\nBei Chen, Fengji Zhang, Anh Nguyen, Daoguang Zan, Zeqi Lin, Jian-Guang Lou, and Weizhu\\nChen. Codet: Code generation with generated tests, 2022.\\nJiaqi Chen, Yuxian Jiang, Jiachen Lu, and Li Zhang. S-agents: self-organizing agents in open-ended\\nenvironment. arXiv preprint , 2024.\\nMark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared\\nKaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri,\\nGretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan,\\nScott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian,\\nClemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fo-\\ntios Chantzis, Elizabeth Barnes, Ariel Herbert-V oss, William Hebgen Guss, Alex Nichol, Alex\\nPaino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders,\\nChristopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec\\nRadford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob Mc-\\nGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large\\nlanguage models trained on code, 2021a.\\n10', start_char_idx=2352, end_char_idx=3884, text_template='{metadata_str}\\n\\n{content}', metadata_template='{key}: {value}', metadata_seperator='\\n'),\n",
       " TextNode(id_='32ce48bd-a65c-4acd-87ac-f2d96acd2a64', embedding=None, metadata={'page_label': '11', 'file_name': 'metagpt.pdf', 'file_path': 'metagpt.pdf', 'file_type': 'application/pdf', 'file_size': 16911937, 'creation_date': '2024-11-25', 'last_modified_date': '2024-06-24'}, excluded_embed_metadata_keys=['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], excluded_llm_metadata_keys=['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], relationships={<NodeRelationship.SOURCE: '1'>: RelatedNodeInfo(node_id='1e71f4a3-378d-4c91-a627-662d5ea88cf4', node_type=<ObjectType.DOCUMENT: '4'>, metadata={'page_label': '11', 'file_name': 'metagpt.pdf', 'file_path': 'metagpt.pdf', 'file_type': 'application/pdf', 'file_size': 16911937, 'creation_date': '2024-11-25', 'last_modified_date': '2024-06-24'}, hash='8f3e556e1f8af697428f095fdd3721319a80876dd4c6cfe2a8817486fd9fac75'), <NodeRelationship.PREVIOUS: '2'>: RelatedNodeInfo(node_id='e1eebe71-b88c-436c-bf9c-fe1d1b53db2c', node_type=<ObjectType.TEXT: '1'>, metadata={'page_label': '10', 'file_name': 'metagpt.pdf', 'file_path': 'metagpt.pdf', 'file_type': 'application/pdf', 'file_size': 16911937, 'creation_date': '2024-11-25', 'last_modified_date': '2024-06-24'}, hash='54c65eebc48c7aa3937573afa29f89cbc85c88d3b4cac58f438547558894aaf6'), <NodeRelationship.NEXT: '3'>: RelatedNodeInfo(node_id='7328a1d9-fb35-4960-a18b-99fd64c94070', node_type=<ObjectType.TEXT: '1'>, metadata={}, hash='d17fd1c20c9c28b8ca87f1cac0b081fef1134d3918569c5bc4391f4bf6a0bf11')}, text='Preprint\\nWeize Chen, Yusheng Su, Jingwei Zuo, Cheng Yang, Chenfei Yuan, Chen Qian, Chi-Min Chan,\\nYujia Qin, Yaxi Lu, Ruobing Xie, Zhiyuan Liu, Maosong Sun, and Jie Zhou. Agentverse: Facili-\\ntating multi-agent collaboration and exploring emergent behaviors in agents, 2023.\\nXinyun Chen, Chang Liu, and Dawn Song. Execution-guided neural program synthesis. In ICLR ,\\n2018.\\nXinyun Chen, Dawn Song, and Yuandong Tian. Latent execution for neural program synthesis\\nbeyond domain-specific languages. NeurIPS , 2021b.\\nAakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam\\nRoberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh,\\nKensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam\\nShazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James\\nBradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Lev-\\nskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin\\nRobinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret\\nZoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick,\\nAndrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica\\nMoreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Bren-\\nnan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas\\nEck, Jeff Dean, Slav Petrov, and Noah Fiedel. Palm: Scaling language modeling with pathways,\\n2022.\\nT. DeMarco and T.R. Lister. Peopleware: Productive Projects and Teams . Addison-Wesley, 2013.\\nURLhttps://books.google.co.uk/books?id=DVlsAQAAQBAJ .\\nYihong Dong, Xue Jiang, Zhi Jin, and Ge Li. Self-collaboration code generation via chatgpt. arXiv\\npreprint , 2023.\\nYilun Du, Shuang Li, Antonio Torralba, Joshua B. Tenenbaum, and Igor Mordatch. Improving\\nfactuality and reasoning in language models through multiagent debate, 2023.\\nYanai Elazar, Nora Kassner, Shauli Ravfogel, Abhilasha Ravichander, Eduard Hovy, Hinrich\\nSch¨utze, and Yoav Goldberg. Measuring and improving consistency in pretrained language mod-\\nels.TACL , 2021.\\nZhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing\\nQin, Ting Liu, Daxin Jiang, et al. Codebert: A pre-trained model for programming and natural\\nlanguages. arXiv preprint , 2020.\\nChrisantha Fernando, Dylan Banarse, Henryk Michalewski, Simon Osindero, and Tim Rockt ¨aschel.\\nPromptbreeder: Self-referential self-improvement via prompt evolution. arXiv preprint , 2023.\\nChelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation\\nof deep networks. In ICML , 2017.\\nDaniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong,\\nWen-tau Yih, Luke Zettlemoyer, and Mike Lewis. Incoder: A generative model for code infilling\\nand synthesis. arXiv preprint , 2022.\\nIrving John Good. Speculations concerning the first ultraintelligent machine. Adv. Comput. , 1965.', start_char_idx=0, end_char_idx=3075, text_template='{metadata_str}\\n\\n{content}', metadata_template='{key}: {value}', metadata_seperator='\\n'),\n",
       " TextNode(id_='7328a1d9-fb35-4960-a18b-99fd64c94070', embedding=None, metadata={'page_label': '11', 'file_name': 'metagpt.pdf', 'file_path': 'metagpt.pdf', 'file_type': 'application/pdf', 'file_size': 16911937, 'creation_date': '2024-11-25', 'last_modified_date': '2024-06-24'}, excluded_embed_metadata_keys=['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], excluded_llm_metadata_keys=['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], relationships={<NodeRelationship.SOURCE: '1'>: RelatedNodeInfo(node_id='1e71f4a3-378d-4c91-a627-662d5ea88cf4', node_type=<ObjectType.DOCUMENT: '4'>, metadata={'page_label': '11', 'file_name': 'metagpt.pdf', 'file_path': 'metagpt.pdf', 'file_type': 'application/pdf', 'file_size': 16911937, 'creation_date': '2024-11-25', 'last_modified_date': '2024-06-24'}, hash='8f3e556e1f8af697428f095fdd3721319a80876dd4c6cfe2a8817486fd9fac75'), <NodeRelationship.PREVIOUS: '2'>: RelatedNodeInfo(node_id='32ce48bd-a65c-4acd-87ac-f2d96acd2a64', node_type=<ObjectType.TEXT: '1'>, metadata={'page_label': '11', 'file_name': 'metagpt.pdf', 'file_path': 'metagpt.pdf', 'file_type': 'application/pdf', 'file_size': 16911937, 'creation_date': '2024-11-25', 'last_modified_date': '2024-06-24'}, hash='ae16eb3aeb4e35661df1299a72de388b8d680bc0471e508ece869bef0114f7c2'), <NodeRelationship.NEXT: '3'>: RelatedNodeInfo(node_id='1c15cbe6-46c0-4e6f-bde3-8909cf4c23d5', node_type=<ObjectType.TEXT: '1'>, metadata={}, hash='e29b5d1cb87f142659fd723de92b1481ef2579707828a7c1678eea72b30bd83d')}, text='Chrisantha Fernando, Dylan Banarse, Henryk Michalewski, Simon Osindero, and Tim Rockt ¨aschel.\\nPromptbreeder: Self-referential self-improvement via prompt evolution. arXiv preprint , 2023.\\nChelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation\\nof deep networks. In ICML , 2017.\\nDaniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong,\\nWen-tau Yih, Luke Zettlemoyer, and Mike Lewis. Incoder: A generative model for code infilling\\nand synthesis. arXiv preprint , 2022.\\nIrving John Good. Speculations concerning the first ultraintelligent machine. Adv. Comput. , 1965.\\nRui Hao, Linmei Hu, Weijian Qi, Qingliu Wu, Yirui Zhang, and Liqiang Nie. Chatllm network:\\nMore brains, more intelligence. arXiv preprint , 2023.\\nS. Hochreiter, A. S. Younger, and P. R. Conwell. Learning to learn using gradient descent. In Lecture\\nNotes on Comp. Sci. 2130, Proc. Intl. Conf. on Artificial Neural Networks (ICANN-2001) , pp. 87–\\n94. Springer: Berlin, Heidelberg, 2001.\\nXue Jiang, Yihong Dong, Lecheng Wang, Qiwei Shang, and Ge Li. Self-planning code generation\\nwith large language model. arXiv preprint , 2023.\\n11', start_char_idx=2433, end_char_idx=3605, text_template='{metadata_str}\\n\\n{content}', metadata_template='{key}: {value}', metadata_seperator='\\n'),\n",
       " TextNode(id_='1c15cbe6-46c0-4e6f-bde3-8909cf4c23d5', embedding=None, metadata={'page_label': '12', 'file_name': 'metagpt.pdf', 'file_path': 'metagpt.pdf', 'file_type': 'application/pdf', 'file_size': 16911937, 'creation_date': '2024-11-25', 'last_modified_date': '2024-06-24'}, excluded_embed_metadata_keys=['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], excluded_llm_metadata_keys=['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], relationships={<NodeRelationship.SOURCE: '1'>: RelatedNodeInfo(node_id='d87e1142-24a1-4109-9c46-aeb56198a91f', node_type=<ObjectType.DOCUMENT: '4'>, metadata={'page_label': '12', 'file_name': 'metagpt.pdf', 'file_path': 'metagpt.pdf', 'file_type': 'application/pdf', 'file_size': 16911937, 'creation_date': '2024-11-25', 'last_modified_date': '2024-06-24'}, hash='eb3106d7578be935f7a4f3071f2ca5ac666776385f374daf39e95ed4342f5bee'), <NodeRelationship.PREVIOUS: '2'>: RelatedNodeInfo(node_id='7328a1d9-fb35-4960-a18b-99fd64c94070', node_type=<ObjectType.TEXT: '1'>, metadata={'page_label': '11', 'file_name': 'metagpt.pdf', 'file_path': 'metagpt.pdf', 'file_type': 'application/pdf', 'file_size': 16911937, 'creation_date': '2024-11-25', 'last_modified_date': '2024-06-24'}, hash='6530b28c140f1f034a9f7938c3a7dc9877aac3ad2925939c98d4c9e9353756cb'), <NodeRelationship.NEXT: '3'>: RelatedNodeInfo(node_id='b0fb5d25-ffaf-46fe-93a1-9dd946c60d1e', node_type=<ObjectType.TEXT: '1'>, metadata={}, hash='35dbd3c1cb3ac2071303e2261b93aede4aba6937fbaf3fdd59b451388a4e42ea')}, text='Preprint\\nGuohao Li, Hasan Abed Al Kader Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem.\\nCamel: Communicative agents for” mind” exploration of large scale language model society.\\narXiv preprint , 2023.\\nYujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, R ´emi Leblond, Tom\\nEccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. Competition-level code generation\\nwith alphacode. Science , 2022.\\nTian Liang, Zhiwei He, Wenxiang Jiao, Xing Wang, Yan Wang, Rui Wang, Yujiu Yang, Zhaopeng\\nTu, and Shuming Shi. Encouraging divergent thinking in large language models through multi-\\nagent debate. arXiv preprint , 2023.\\nBill Yuchen Lin, Yicheng Fu, Karina Yang, Prithviraj Ammanabrolu, Faeze Brahman, Shiyu Huang,\\nChandra Bhagavatula, Yejin Choi, and Xiang Ren. Swiftsage: A generative agent with fast and\\nslow thinking for complex interactive tasks. arXiv preprint , 2023.\\nRuibo Liu, Ruixin Yang, Chenyan Jia, Ge Zhang, Denny Zhou, Andrew M Dai, Diyi Yang, and\\nSoroush V osoughi. Training socially aligned language models in simulated human society. arXiv\\npreprint , 2023.\\nZiyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing\\nMa, Qingwei Lin, and Daxin Jiang. Wizardcoder: Empowering code large language models with\\nevol-instruct. arXiv preprint , 2023.\\nPotsawee Manakul, Adian Liusie, and Mark JF Gales. Selfcheckgpt: Zero-resource black-box hal-\\nlucination detection for generative large language models. arXiv preprint , 2023.\\nAgile Manifesto. Manifesto for agile software development . Snowbird, UT, 2001.\\nJohn McCarthy. History of lisp. In History of programming languages . 1978.\\nAnsong Ni, Srini Iyer, Dragomir Radev, Veselin Stoyanov, Wen-tau Yih, Sida Wang, and Xi Victoria\\nLin. Lever: Learning to verify language-to-code generation with execution. In ICML , 2023.\\nErik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese,\\nand Caiming Xiong. Codegen: An open large language model for code with multi-turn program\\nsynthesis, 2023.\\nOpenAI. Gpt-4 technical report, 2023.\\nJoon Sung Park, Joseph C O’Brien, Carrie J Cai, Meredith Ringel Morris, Percy Liang, and\\nMichael S Bernstein. Generative agents: Interactive simulacra of human behavior. arXiv preprint ,\\n2023.\\nChen Qian, Xin Cong, Cheng Yang, Weize Chen, Yusheng Su, Juyuan Xu, Zhiyuan Liu, and\\nMaosong Sun. Communicative agents for software development, 2023.\\nBaptiste Rozi `ere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi\\nAdi, Jingyu Liu, Tal Remez, J ´er´emy Rapin, et al. Code llama: Open foundation models for code.\\narXiv preprint , 2023.\\nTimo Schick, Jane Dwivedi-Yu, Roberto Dess `ı, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer,\\nNicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to\\nuse tools. arXiv preprint , 2023.\\nJ. Schmidhuber. A self-referential weight matrix. In Proceedings of the International Conference\\non Artificial Neural Networks, Amsterdam , pp. 446–451. Springer, 1993a.\\nJ. Schmidhuber. G ¨odel machines: self-referential universal problem solvers making provably\\noptimal self-improvements.', start_char_idx=0, end_char_idx=3154, text_template='{metadata_str}\\n\\n{content}', metadata_template='{key}: {value}', metadata_seperator='\\n'),\n",
       " TextNode(id_='b0fb5d25-ffaf-46fe-93a1-9dd946c60d1e', embedding=None, metadata={'page_label': '12', 'file_name': 'metagpt.pdf', 'file_path': 'metagpt.pdf', 'file_type': 'application/pdf', 'file_size': 16911937, 'creation_date': '2024-11-25', 'last_modified_date': '2024-06-24'}, excluded_embed_metadata_keys=['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], excluded_llm_metadata_keys=['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], relationships={<NodeRelationship.SOURCE: '1'>: RelatedNodeInfo(node_id='d87e1142-24a1-4109-9c46-aeb56198a91f', node_type=<ObjectType.DOCUMENT: '4'>, metadata={'page_label': '12', 'file_name': 'metagpt.pdf', 'file_path': 'metagpt.pdf', 'file_type': 'application/pdf', 'file_size': 16911937, 'creation_date': '2024-11-25', 'last_modified_date': '2024-06-24'}, hash='eb3106d7578be935f7a4f3071f2ca5ac666776385f374daf39e95ed4342f5bee'), <NodeRelationship.PREVIOUS: '2'>: RelatedNodeInfo(node_id='1c15cbe6-46c0-4e6f-bde3-8909cf4c23d5', node_type=<ObjectType.TEXT: '1'>, metadata={'page_label': '12', 'file_name': 'metagpt.pdf', 'file_path': 'metagpt.pdf', 'file_type': 'application/pdf', 'file_size': 16911937, 'creation_date': '2024-11-25', 'last_modified_date': '2024-06-24'}, hash='499d34caaa0452b7dd0adb64005abb0807c2d2ee1e6b715a89a4291557d5177c'), <NodeRelationship.NEXT: '3'>: RelatedNodeInfo(node_id='2b38798d-e6d8-40d7-a361-a4ff816b0fa3', node_type=<ObjectType.TEXT: '1'>, metadata={}, hash='deff53f820ea3c5f8a0b0d722e187087cd15a6ad68eb200072b892de4b20f8b0')}, text='Code llama: Open foundation models for code.\\narXiv preprint , 2023.\\nTimo Schick, Jane Dwivedi-Yu, Roberto Dess `ı, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer,\\nNicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to\\nuse tools. arXiv preprint , 2023.\\nJ. Schmidhuber. A self-referential weight matrix. In Proceedings of the International Conference\\non Artificial Neural Networks, Amsterdam , pp. 446–451. Springer, 1993a.\\nJ. Schmidhuber. G ¨odel machines: self-referential universal problem solvers making provably\\noptimal self-improvements. Technical Report IDSIA-19-03, arXiv:cs.LO/0309048 v3, IDSIA,\\nManno-Lugano, Switzerland, December 2003.\\nJ. Schmidhuber. G ¨odel machines: Fully self-referential optimal universal self-improvers. In B. Go-\\nertzel and C. Pennachin (eds.), Artificial General Intelligence , pp. 199–226. Springer Verlag,\\n2006. Variant available as arXiv:cs.LO/0309048.\\n12', start_char_idx=2577, end_char_idx=3505, text_template='{metadata_str}\\n\\n{content}', metadata_template='{key}: {value}', metadata_seperator='\\n'),\n",
       " TextNode(id_='2b38798d-e6d8-40d7-a361-a4ff816b0fa3', embedding=None, metadata={'page_label': '13', 'file_name': 'metagpt.pdf', 'file_path': 'metagpt.pdf', 'file_type': 'application/pdf', 'file_size': 16911937, 'creation_date': '2024-11-25', 'last_modified_date': '2024-06-24'}, excluded_embed_metadata_keys=['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], excluded_llm_metadata_keys=['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], relationships={<NodeRelationship.SOURCE: '1'>: RelatedNodeInfo(node_id='9b52a884-e751-4588-a635-16e391046305', node_type=<ObjectType.DOCUMENT: '4'>, metadata={'page_label': '13', 'file_name': 'metagpt.pdf', 'file_path': 'metagpt.pdf', 'file_type': 'application/pdf', 'file_size': 16911937, 'creation_date': '2024-11-25', 'last_modified_date': '2024-06-24'}, hash='611466ee0bd48c2f4f315dd93980953535431a01de333799b656294c2ee2442d'), <NodeRelationship.PREVIOUS: '2'>: RelatedNodeInfo(node_id='b0fb5d25-ffaf-46fe-93a1-9dd946c60d1e', node_type=<ObjectType.TEXT: '1'>, metadata={'page_label': '12', 'file_name': 'metagpt.pdf', 'file_path': 'metagpt.pdf', 'file_type': 'application/pdf', 'file_size': 16911937, 'creation_date': '2024-11-25', 'last_modified_date': '2024-06-24'}, hash='cb515945e708e2db1bf555cc1b859566bc4ecfe0df3c06345597959ff542dbbe'), <NodeRelationship.NEXT: '3'>: RelatedNodeInfo(node_id='649d9862-4c17-44fd-92e9-56558888a5ae', node_type=<ObjectType.TEXT: '1'>, metadata={}, hash='e6817a11c611d868bfa9687dcbd3cf1d0eaf3cddd57126de3b3a0bc68c169b8d')}, text='Preprint\\nJ. Schmidhuber. Ultimate cognition `a laG¨odel. Cognitive Computation , 1(2):177–193, 2009.\\nJ¨urgen Schmidhuber. Evolutionary principles in self-referential learning, or on learning how to\\nlearn: the meta-meta-... hook . PhD thesis, 1987.\\nJ¨urgen Schmidhuber. A ‘self-referential’weight matrix. In ICANN’93: Proceedings of the Interna-\\ntional Conference on Artificial Neural Networks Amsterdam, The Netherlands 13–16 September\\n1993 3 , 1993b.\\nJ¨urgen Schmidhuber. On learning to think: Algorithmic information theory for novel combinations\\nof reinforcement learning controllers and recurrent neural world models. arXiv preprint , 2015.\\nJ¨urgen Schmidhuber, Jieyu Zhao, and Nicol N Schraudolph. Reinforcement learning with self-\\nmodifying policies. In Learning to learn . 1998.\\nNoah Shinn, Beck Labash, and Ashwin Gopinath. Reflexion: an autonomous agent with dynamic\\nmemory and self-reflection. arXiv preprint , 2023.\\nMarta Skreta, Naruki Yoshikawa, Sebastian Arellano-Rubach, Zhi Ji, Lasse Bjørn Kristensen,\\nKourosh Darvish, Al ´an Aspuru-Guzik, Florian Shkurti, and Animesh Garg. Errors are useful\\nprompts: Instruction guided task programming with verifier-assisted iterative prompting. arXiv\\npreprint , 2023.\\nElliot Soloway. Learning to program =learning to construct mechanisms and explanations. Com-\\nmunications of the ACM , 1986.\\nYashar Talebirad and Amirhossein Nadiri. Multi-agent collaboration: Harnessing the power of\\nintelligent llm agents, 2023.\\nTorantulino et al. Auto-gpt. https://github.com/Significant-Gravitas/\\nAuto-GPT , 2023.\\nR. J. Waldinger and R. C. T. Lee. PROW: a step toward automatic program writing. In D. E. Walker\\nand L. M. Norton (eds.), Proceedings of the 1st International Joint Conference on Artificial Intel-\\nligence (IJCAI) , 1969.\\nGuanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan,\\nand Anima Anandkumar. V oyager: An open-ended embodied agent with large language models.\\narXiv preprint , 2023a.\\nLei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai\\nTang, Xu Chen, Yankai Lin, et al. A survey on large language model based autonomous agents.\\narXiv preprint , 2023b.\\nXuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdh-\\nery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models.\\narXiv preprint , 2022.\\nZhenhailong Wang, Shaoguang Mao, Wenshan Wu, Tao Ge, Furu Wei, and Heng Ji. Unleashing\\ncognitive synergy in large language models: A task-solving agent through multi-persona self-\\ncollaboration. arXiv preprint , 2023c.\\nJason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny\\nZhou, et al. Chain-of-thought prompting elicits reasoning in large language models. NeurIPS ,\\n2022.\\nMichael Wooldridge and Nicholas R. Jennings. Pitfalls of agent-oriented development. In Pro-\\nceedings of the Second International Conference on Autonomous Agents , 1998. URL https:\\n//doi.org/10.1145/280765.280867 .\\nShunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao.\\nReact: Synergizing reasoning and acting in language models. arXiv preprint , 2022.\\nEric Zelikman, Eliana Lorch, Lester Mackey, and Adam Tauman Kalai. Self-taught optimizer (stop):\\nRecursively self-improving code generation. arXiv preprint , 2023.\\n13', start_char_idx=0, end_char_idx=3357, text_template='{metadata_str}\\n\\n{content}', metadata_template='{key}: {value}', metadata_seperator='\\n'),\n",
       " TextNode(id_='649d9862-4c17-44fd-92e9-56558888a5ae', embedding=None, metadata={'page_label': '14', 'file_name': 'metagpt.pdf', 'file_path': 'metagpt.pdf', 'file_type': 'application/pdf', 'file_size': 16911937, 'creation_date': '2024-11-25', 'last_modified_date': '2024-06-24'}, excluded_embed_metadata_keys=['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], excluded_llm_metadata_keys=['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], relationships={<NodeRelationship.SOURCE: '1'>: RelatedNodeInfo(node_id='60dd89f1-2c56-4621-b297-e691d658b551', node_type=<ObjectType.DOCUMENT: '4'>, metadata={'page_label': '14', 'file_name': 'metagpt.pdf', 'file_path': 'metagpt.pdf', 'file_type': 'application/pdf', 'file_size': 16911937, 'creation_date': '2024-11-25', 'last_modified_date': '2024-06-24'}, hash='095dfd61de4bdb5254454f7ef7bbc3be36683c46a9008e7ee84abc8afe4ca858'), <NodeRelationship.PREVIOUS: '2'>: RelatedNodeInfo(node_id='2b38798d-e6d8-40d7-a361-a4ff816b0fa3', node_type=<ObjectType.TEXT: '1'>, metadata={'page_label': '13', 'file_name': 'metagpt.pdf', 'file_path': 'metagpt.pdf', 'file_type': 'application/pdf', 'file_size': 16911937, 'creation_date': '2024-11-25', 'last_modified_date': '2024-06-24'}, hash='611466ee0bd48c2f4f315dd93980953535431a01de333799b656294c2ee2442d'), <NodeRelationship.NEXT: '3'>: RelatedNodeInfo(node_id='6b735536-2bab-4345-9768-70e00bcf9cad', node_type=<ObjectType.TEXT: '1'>, metadata={}, hash='b2016d296fdee721d345237dfc7f6c814177a9cc09f95e0030bfcd39c8602eee')}, text='Preprint\\nHongxin Zhang, Weihua Du, Jiaming Shan, Qinhong Zhou, Yilun Du, Joshua B Tenenbaum, Tian-\\nmin Shu, and Chuang Gan. Building cooperative embodied agents modularly with large language\\nmodels. arXiv preprint , 2023.\\nXufeng Zhao, Mengdi Li, Cornelius Weber, Muhammad Burhan Hafez, and Stefan Wermter. Chat\\nwith the environment: Interactive multimodal perception using large language models. arXiv\\npreprint , 2023.\\nQinkai Zheng, Xiao Xia, Xu Zou, Yuxiao Dong, Shan Wang, Yufei Xue, Zihan Wang, Lei Shen,\\nAndi Wang, Yang Li, Teng Su, Zhilin Yang, and Jie Tang. Codegeex: A pre-trained model for\\ncode generation with multilingual evaluations on humaneval-x, 2023.\\nShuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng,\\nYonatan Bisk, Daniel Fried, Uri Alon, et al. Webarena: A realistic web environment for building\\nautonomous agents. arXiv preprint , 2023.\\nMingchen Zhuge, Haozhe Liu, Francesco Faccio, Dylan R Ashley, R ´obert Csord ´as, Anand\\nGopalakrishnan, Abdullah Hamdi, Hasan Abed Al Kader Hammoud, Vincent Herrmann, Kazuki\\nIrie, et al. Mindstorms in natural language-based societies of mind. arXiv preprint , 2023.\\n14', start_char_idx=0, end_char_idx=1160, text_template='{metadata_str}\\n\\n{content}', metadata_template='{key}: {value}', metadata_seperator='\\n'),\n",
       " TextNode(id_='6b735536-2bab-4345-9768-70e00bcf9cad', embedding=None, metadata={'page_label': '15', 'file_name': 'metagpt.pdf', 'file_path': 'metagpt.pdf', 'file_type': 'application/pdf', 'file_size': 16911937, 'creation_date': '2024-11-25', 'last_modified_date': '2024-06-24'}, excluded_embed_metadata_keys=['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], excluded_llm_metadata_keys=['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], relationships={<NodeRelationship.SOURCE: '1'>: RelatedNodeInfo(node_id='6beccc68-e1ec-4ba5-9688-bd8bcfb07e03', node_type=<ObjectType.DOCUMENT: '4'>, metadata={'page_label': '15', 'file_name': 'metagpt.pdf', 'file_path': 'metagpt.pdf', 'file_type': 'application/pdf', 'file_size': 16911937, 'creation_date': '2024-11-25', 'last_modified_date': '2024-06-24'}, hash='55cf295d468aef6bf1856b261ebe93ecf90430f58231ce15f167a9a071253be5'), <NodeRelationship.PREVIOUS: '2'>: RelatedNodeInfo(node_id='649d9862-4c17-44fd-92e9-56558888a5ae', node_type=<ObjectType.TEXT: '1'>, metadata={'page_label': '14', 'file_name': 'metagpt.pdf', 'file_path': 'metagpt.pdf', 'file_type': 'application/pdf', 'file_size': 16911937, 'creation_date': '2024-11-25', 'last_modified_date': '2024-06-24'}, hash='095dfd61de4bdb5254454f7ef7bbc3be36683c46a9008e7ee84abc8afe4ca858'), <NodeRelationship.NEXT: '3'>: RelatedNodeInfo(node_id='ce8efe4e-47ae-4f7e-b534-452ee49a5507', node_type=<ObjectType.TEXT: '1'>, metadata={}, hash='86c3e2056623ef25f7b7bfc460f5546c576a78dfdf4e058c203dc2ddb25c61f2')}, text='Preprint\\nA O UTLOOK\\nA.1 S ELF-IMPROVEMENT MECHANISMS\\nOne limitation of the MetaGPT version in the main text of this paper is that each software project is\\nexecuted independently. However, through active teamwork, a software development team should\\nlearn from the experience gained by developing each project, thus becoming more compatible and\\nsuccessful over time.\\nThis is somewhat related to the idea of recursive self-improvement, first informally proposed in\\n1965 (Good, 1965), with first concrete implementations since 1987 (Schmidhuber, 1987; 1993b;\\nSchmidhuber et al., 1998), culminating in the concept of mathematically optimal self-referential\\nself-improvers (Schmidhuber, 2003; 2009). Generally speaking, a system should learn from experi-\\nence in the real world, and meta-learn better learning algorithms from experiences of learning, and\\nmeta-meta-learn better meta-learning algorithms from experiences of meta-learning, etc., without\\nany limitations except those of computability and physics.\\nMore recent, somewhat related work leverages the reasoning ability of Large Language Models\\n(LLMs) and recursively improves prompts of LLMs, to improve performance on certain downstream\\ntasks (Fernando et al., 2023; Zelikman et al., 2023), analogous to the adaptive prompt engineer of\\n2015 (Schmidhuber, 2015) where one neural network learns to generate sequence of queries or\\nprompts for another pre-trained neural network whose answers may help the first network to learn\\nnew tasks more quickly.\\nIn our present work, we also explore a self-referential mechanism that recursively modifies the con-\\nstraint prompts of agents based on information they observe during software development. Our\\ninitial implementation works as follows. Prior to each project, every agent in the software company\\nreviews previous feedback and makes necessary adjustments to their constraint prompts. This en-\\nables them to continuously learn from past project experiences and enhance the overall multi-agent\\nsystem by improving each individual in the company. We first establish a handover feedback action\\nfor each agent. This action is responsible for critically summarizing the information received dur-\\ning the development of previous projects and integrating this information in an updated constraint\\nprompt. The summarized information is stored in long-term memory such that it can be inherited\\nby future constraint prompt updates. When initiating a new project, each agent starts with a react\\naction. Each agent evaluates the received feedback and summarizes how they can improve in a\\nconstraint prompt.\\nOne current limitation is that these summary-based optimizations only modify constraints in the\\nspecialization of roles (Sec. 3.1) rather than structured communication interfaces in communication\\nprotocols (Sec. 3.2). Future advancements are yet to be explored.\\nA.2 M ULTI -AGENT ECONOMIES\\nIn real-world teamwork, the interaction processes are often not hardcoded. For example, in a soft-\\nware company, the collaboration SOP may change dynamically.\\nOne implementation of such self-organization is discussed in the paper on a “Natural Language-\\nBased Society of Mind” (NLSOM) (Zhuge et al., 2023), which introduced the idea of an “Economy\\nof Minds” (EOM), a Reinforcement Learning (RL) framework for societies of LLMs and other\\nagents. Instead of using standard RL techniques to optimize the total reward of the system through\\nmodifications of neural network parameters, EOMs use the principles of supply and demand in free\\nmarkets to assign credit (money) to those agents that contribute to economic success (reward).\\nThe recent agent-based platform of DeepWisdom (AgentStore4) is compatible with the credit as-\\nsignment concept of EOMs. Each agent in AgentStore provides a list of services with corresponding\\ncosts. A convenient API is provided so that human users or agents in the platform can easily pur-\\nchase services from other agents to accomplish their services. Figure 6 displays the User Interface\\n(UI) of AgentStore, where various agents with different skills are showcased. Besides, individual\\ndevelopers can participate in building new agents and enable collaborative development within the\\ncommunity. Specifically, AgentStore allows users to subscribe to agents according to their demands\\n4http://beta.deepwisdom.ai\\n15', start_char_idx=0, end_char_idx=4326, text_template='{metadata_str}\\n\\n{content}', metadata_template='{key}: {value}', metadata_seperator='\\n'),\n",
       " TextNode(id_='ce8efe4e-47ae-4f7e-b534-452ee49a5507', embedding=None, metadata={'page_label': '16', 'file_name': 'metagpt.pdf', 'file_path': 'metagpt.pdf', 'file_type': 'application/pdf', 'file_size': 16911937, 'creation_date': '2024-11-25', 'last_modified_date': '2024-06-24'}, excluded_embed_metadata_keys=['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], excluded_llm_metadata_keys=['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], relationships={<NodeRelationship.SOURCE: '1'>: RelatedNodeInfo(node_id='691af0cc-63af-42f2-83e6-312a3f7a5308', node_type=<ObjectType.DOCUMENT: '4'>, metadata={'page_label': '16', 'file_name': 'metagpt.pdf', 'file_path': 'metagpt.pdf', 'file_type': 'application/pdf', 'file_size': 16911937, 'creation_date': '2024-11-25', 'last_modified_date': '2024-06-24'}, hash='a7f7578ea2b1c3071389838a88d98205df3a1ff48c5b1d42bade94871b1b63ff'), <NodeRelationship.PREVIOUS: '2'>: RelatedNodeInfo(node_id='6b735536-2bab-4345-9768-70e00bcf9cad', node_type=<ObjectType.TEXT: '1'>, metadata={'page_label': '15', 'file_name': 'metagpt.pdf', 'file_path': 'metagpt.pdf', 'file_type': 'application/pdf', 'file_size': 16911937, 'creation_date': '2024-11-25', 'last_modified_date': '2024-06-24'}, hash='55cf295d468aef6bf1856b261ebe93ecf90430f58231ce15f167a9a071253be5'), <NodeRelationship.NEXT: '3'>: RelatedNodeInfo(node_id='3670da62-e7f4-43b7-9387-210775cc6b0c', node_type=<ObjectType.TEXT: '1'>, metadata={}, hash='12fab44c726150106a62a70bc02ea8848b1ad42b60da9d048295f746573592b1')}, text='Preprint\\nand pay according to their usage. Moreover, users can purchase additional capabilities to expand the\\nplug-and-play functions of their existing agents. This allows users to gradually upgrade their agents.\\nWithin the MetaGPT framework, AgentStore can support the collaboration of various agents. Users\\ncan collect several agents together to carry out more complex tasks or projects, and all the agents\\nshare and comply with development and communication protocols defined in MetaGPT.\\nFigure 6: AgentStore is a platform dedicated to serving users in the creation and development of\\nagents within the MetaGPT framework. This platform provides users with an operational interface,\\nallowing users to easily manage a variety of agents with different emotions, personalities, and capa-\\nbilities for specific tasks.\\n16', start_char_idx=0, end_char_idx=818, text_template='{metadata_str}\\n\\n{content}', metadata_template='{key}: {value}', metadata_seperator='\\n'),\n",
       " TextNode(id_='3670da62-e7f4-43b7-9387-210775cc6b0c', embedding=None, metadata={'page_label': '17', 'file_name': 'metagpt.pdf', 'file_path': 'metagpt.pdf', 'file_type': 'application/pdf', 'file_size': 16911937, 'creation_date': '2024-11-25', 'last_modified_date': '2024-06-24'}, excluded_embed_metadata_keys=['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], excluded_llm_metadata_keys=['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], relationships={<NodeRelationship.SOURCE: '1'>: RelatedNodeInfo(node_id='689534c4-a317-43f1-b672-3a79b41fa255', node_type=<ObjectType.DOCUMENT: '4'>, metadata={'page_label': '17', 'file_name': 'metagpt.pdf', 'file_path': 'metagpt.pdf', 'file_type': 'application/pdf', 'file_size': 16911937, 'creation_date': '2024-11-25', 'last_modified_date': '2024-06-24'}, hash='d0d50698df3e8e59f5ef90ec22bd3ee03f93cf42c77905fbec081205342e97ac'), <NodeRelationship.PREVIOUS: '2'>: RelatedNodeInfo(node_id='ce8efe4e-47ae-4f7e-b534-452ee49a5507', node_type=<ObjectType.TEXT: '1'>, metadata={'page_label': '16', 'file_name': 'metagpt.pdf', 'file_path': 'metagpt.pdf', 'file_type': 'application/pdf', 'file_size': 16911937, 'creation_date': '2024-11-25', 'last_modified_date': '2024-06-24'}, hash='a7f7578ea2b1c3071389838a88d98205df3a1ff48c5b1d42bade94871b1b63ff'), <NodeRelationship.NEXT: '3'>: RelatedNodeInfo(node_id='a640fae4-ee87-4e98-8be5-55733bfc3842', node_type=<ObjectType.TEXT: '1'>, metadata={}, hash='a1db75712a39b0349735dc2ca85c6eabe1caa4c5d2e73bc5b46dcf09ba5fc94d')}, text='Preprint\\nB A D EMO OF THE EXECUTION\\nIn this section, we outline the complete process of software development using MetaGPT. It begins\\nwith a user’s input command (as shown in Appendix B.1) and ends with software designed according\\nto the user’s specifications.\\nB.1 U SERINPUT\\nUpon receiving an instruction from the user, MetaGPT collaborates with a professional development\\nteam to fulfill the task. Here is a demo of user input:\\nUser input\\n1write a python3 GUI app such that you can draw an image with it\\nB.2 M ETAGPT DEVELOPMENT PROCESS\\nNow we provide a step-by-step explanation of the standardized output process for each agent.\\nProduct Manager The Product Manager generates a Product Requirement Document (PRD), as\\ndetailed in the specified documentation. This document encompasses goals, user stories, competitive\\nanalysis, requirement analysis and requirement pool. Additionally, a competitive quadrant chart is\\nproduced (see Figure 7). Subsequently, these documents and charts are handed over to the architect\\nfor system design.\\nProduct Requirement Document\\n1## Original Requirements\\n2The boss requires a Python3 GUI color meter that outputs the RGB values when it moves.\\n3\\n4## Product Goals\\n5‘‘‘python\\n6[\\n7 \"Create a user-friendly GUI color meter\",\\n8 \"Ensure the color meter accurately outputs RGB values\",\\n9 \"Ensure the color meter updates RGB values in real-time as it moves\"\\n10]\\n11‘‘‘\\n12\\n13## User Stories\\n14‘‘‘python\\n15[\\n16 \"As a user, I want to select any color on the screen, so that I can get its RGB values\",\\n17 \"As a user, I want the RGB values to update in real-time as I move the color meter, so that I\\n,→can get accurate color information\",\\n18 \"As a user, I want a simple and intuitive interface, so that I can easily use the color meter\"\\n19]\\n20‘‘‘\\n21\\n22## Competitive Analysis\\n23‘‘‘python\\n24[\\n25 \"Color Cop: A popular color picker for Windows. It’s simple and easy to use, but lacks real-\\n,→time RGB value updates\",\\n26 \"Just Color Picker: Offers real-time color picking and multiple color code formats. However,\\n,→its interface is cluttered and not very user-friendly\",\\n27 \"Instant Eyedropper: A simple color picker that lives in the system tray. It’s convenient but\\n,→lacks advanced features\",\\n28 \"ColorZilla: A color picker extension for browsers. It’s convenient for web design but not\\n,→suitable for desktop applications\",\\n29 \"ColorPic: Offers color palettes and mixer tools. It’s feature-rich but can be overwhelming\\n,→for simple tasks\"\\n30]\\n31‘‘‘\\n17', start_char_idx=0, end_char_idx=2475, text_template='{metadata_str}\\n\\n{content}', metadata_template='{key}: {value}', metadata_seperator='\\n'),\n",
       " TextNode(id_='a640fae4-ee87-4e98-8be5-55733bfc3842', embedding=None, metadata={'page_label': '18', 'file_name': 'metagpt.pdf', 'file_path': 'metagpt.pdf', 'file_type': 'application/pdf', 'file_size': 16911937, 'creation_date': '2024-11-25', 'last_modified_date': '2024-06-24'}, excluded_embed_metadata_keys=['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], excluded_llm_metadata_keys=['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], relationships={<NodeRelationship.SOURCE: '1'>: RelatedNodeInfo(node_id='b2d51d18-762d-455c-950c-97774fa89796', node_type=<ObjectType.DOCUMENT: '4'>, metadata={'page_label': '18', 'file_name': 'metagpt.pdf', 'file_path': 'metagpt.pdf', 'file_type': 'application/pdf', 'file_size': 16911937, 'creation_date': '2024-11-25', 'last_modified_date': '2024-06-24'}, hash='abe5e76e3f6d2108f50923d0b558a742edd8b68fe2624ae86b6ff05e966ce4bc'), <NodeRelationship.PREVIOUS: '2'>: RelatedNodeInfo(node_id='3670da62-e7f4-43b7-9387-210775cc6b0c', node_type=<ObjectType.TEXT: '1'>, metadata={'page_label': '17', 'file_name': 'metagpt.pdf', 'file_path': 'metagpt.pdf', 'file_type': 'application/pdf', 'file_size': 16911937, 'creation_date': '2024-11-25', 'last_modified_date': '2024-06-24'}, hash='d0d50698df3e8e59f5ef90ec22bd3ee03f93cf42c77905fbec081205342e97ac'), <NodeRelationship.NEXT: '3'>: RelatedNodeInfo(node_id='8f5694c1-2784-4cfa-8948-7a0da5418d44', node_type=<ObjectType.TEXT: '1'>, metadata={}, hash='5714c7acbb72b9a22f80a6f2f8b2c107448273543f47182ba82874987549a8fb')}, text='Preprint\\nProduct Requirement Document\\n1\\n2## Requirement Analysis\\n3The product should be a Python3 GUI application that serves as a color meter. It should allow\\n,→users to select any color on the screen and display the corresponding RGB values. The RGB\\n,→ values should update in real-time as the color meter moves. The interface should be\\n,→simple and user-friendly.\\n4\\n5## Requirement Pool\\n6‘‘‘python\\n7[\\n8 (\"Design a user-friendly GUI for the color meter\", \"P0\"),\\n9 (\"Implement color selection functionality\", \"P0\"),\\n10 (\"Display RGB values of the selected color\", \"P0\"),\\n11 (\"Update RGB values in real-time as the color meter moves\", \"P0\"),\\n12 (\"Test the application for accuracy and performance\", \"P1\")\\n13]\\n14‘‘‘\\n15\\n16## UI Design draft\\n17The UI should be minimalistic and intuitive. It should primarily consist of a color meter, which\\n,→could be a magnified area of the screen that the user can move around. The RGB values of\\n,→the currently selected color should be prominently displayed, updating in real-time as\\n,→the color meter moves. The layout should be clean and uncluttered, with a focus on\\n,→functionality and ease of use.\\n18\\n19## Anything UNCLEAR\\n20There are no unclear points.\\nFigure 7: The quadrant chart for the “Drawing App” generated by the product manager agent.\\n18', start_char_idx=0, end_char_idx=1285, text_template='{metadata_str}\\n\\n{content}', metadata_template='{key}: {value}', metadata_seperator='\\n'),\n",
       " TextNode(id_='8f5694c1-2784-4cfa-8948-7a0da5418d44', embedding=None, metadata={'page_label': '19', 'file_name': 'metagpt.pdf', 'file_path': 'metagpt.pdf', 'file_type': 'application/pdf', 'file_size': 16911937, 'creation_date': '2024-11-25', 'last_modified_date': '2024-06-24'}, excluded_embed_metadata_keys=['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], excluded_llm_metadata_keys=['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], relationships={<NodeRelationship.SOURCE: '1'>: RelatedNodeInfo(node_id='e847cb3d-b311-4252-994f-1907b68d5383', node_type=<ObjectType.DOCUMENT: '4'>, metadata={'page_label': '19', 'file_name': 'metagpt.pdf', 'file_path': 'metagpt.pdf', 'file_type': 'application/pdf', 'file_size': 16911937, 'creation_date': '2024-11-25', 'last_modified_date': '2024-06-24'}, hash='336d4ff1aa8ce1e43232f52c27c6744251f10db696ad26ff3cf3dfe34b410556'), <NodeRelationship.PREVIOUS: '2'>: RelatedNodeInfo(node_id='a640fae4-ee87-4e98-8be5-55733bfc3842', node_type=<ObjectType.TEXT: '1'>, metadata={'page_label': '18', 'file_name': 'metagpt.pdf', 'file_path': 'metagpt.pdf', 'file_type': 'application/pdf', 'file_size': 16911937, 'creation_date': '2024-11-25', 'last_modified_date': '2024-06-24'}, hash='abe5e76e3f6d2108f50923d0b558a742edd8b68fe2624ae86b6ff05e966ce4bc'), <NodeRelationship.NEXT: '3'>: RelatedNodeInfo(node_id='f26547d5-cee7-46ed-b0e3-e5bc2b1f758e', node_type=<ObjectType.TEXT: '1'>, metadata={}, hash='d6b034dc323d360be5bae0b4b30207ea6f4b96764cc4e1b3d1c295c70f06b6bb')}, text='Preprint\\nArchitect Based on the requirements in PRD, the Architect agent devises technical specifications\\nincluding system architecture diagrams and interface definitions. Initially, the Architect defines the\\noverarching technical trajectory. Subsequently, the project’s architecture, including files, classes\\n(Figure 8) and the sequence flow chart (Figure 12), is designed. The Architect’s documentation is\\nthen given to the project manager for task allocation and execution.\\nSystem design\\n1## Implementation approach\\n2We will use Python’s Tkinter library to create the GUI for the color meter. Tkinter is a standard\\n,→Python interface to the Tk GUI toolkit and is included with most Python installations. It\\n,→ is simple to use and powerful enough for our needs.\\n3\\n4The color selection functionality will be implemented using the PIL (Pillow) library, which allows\\n,→ us to capture the screen’s pixel data. By moving the mouse, we can get the RGB values of\\n,→ the pixel under the mouse cursor in real-time.\\n5\\n6The RGB values will be displayed on the GUI using Tkinter’s Label widget. We will use Python’s\\n,→threading library to update the RGB values in real-time as the color meter moves.\\n7\\n8The application will be tested for accuracy and performance using Python’s unittest library.\\n9\\n10## Python package name\\n11‘‘‘python\\n12\"color_meter\"\\n13‘‘‘\\n14\\n15## File list\\n16‘‘‘python\\n17[\\n18 \"main.py\",\\n19 \"color_picker.py\",\\n20 \"gui.py\",\\n21 \"tests.py\"\\n22]\\n23‘‘‘\\n24\\n25## Anything UNCLEAR\\n26The requirement is clear to me.\\nFigure 8: Data structures and interface definitions for the “Drawing App” generated by the architect\\nagent.\\nProject Manager The Project Manager breaks down the project into a task list. Furthermore, each\\ncode file is analyzed based on its intended functionality and then treated as a separate task assigned\\nto Engineers.\\n19', start_char_idx=0, end_char_idx=1837, text_template='{metadata_str}\\n\\n{content}', metadata_template='{key}: {value}', metadata_seperator='\\n'),\n",
       " TextNode(id_='f26547d5-cee7-46ed-b0e3-e5bc2b1f758e', embedding=None, metadata={'page_label': '20', 'file_name': 'metagpt.pdf', 'file_path': 'metagpt.pdf', 'file_type': 'application/pdf', 'file_size': 16911937, 'creation_date': '2024-11-25', 'last_modified_date': '2024-06-24'}, excluded_embed_metadata_keys=['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], excluded_llm_metadata_keys=['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], relationships={<NodeRelationship.SOURCE: '1'>: RelatedNodeInfo(node_id='a960eba4-be2c-473c-84c0-5b3204d0ea11', node_type=<ObjectType.DOCUMENT: '4'>, metadata={'page_label': '20', 'file_name': 'metagpt.pdf', 'file_path': 'metagpt.pdf', 'file_type': 'application/pdf', 'file_size': 16911937, 'creation_date': '2024-11-25', 'last_modified_date': '2024-06-24'}, hash='cb741a963667fada46987dbb295f76acec2a86b01bb8969c681ca10b25c9c8e0'), <NodeRelationship.PREVIOUS: '2'>: RelatedNodeInfo(node_id='8f5694c1-2784-4cfa-8948-7a0da5418d44', node_type=<ObjectType.TEXT: '1'>, metadata={'page_label': '19', 'file_name': 'metagpt.pdf', 'file_path': 'metagpt.pdf', 'file_type': 'application/pdf', 'file_size': 16911937, 'creation_date': '2024-11-25', 'last_modified_date': '2024-06-24'}, hash='336d4ff1aa8ce1e43232f52c27c6744251f10db696ad26ff3cf3dfe34b410556'), <NodeRelationship.NEXT: '3'>: RelatedNodeInfo(node_id='b645a3e6-de6e-44c9-96a1-ba682315f3e2', node_type=<ObjectType.TEXT: '1'>, metadata={}, hash='3256dc6a8060be5e13020c1b0a0abe9697192d7b643ddc94983191f89704d5d1')}, text='Preprint\\nFigure 9: The program call flow for the “Drawing App” generated by the architect agent.\\nTasks\\n1## Required Python third-party packages\\n2\\n3‘‘‘python\\n4\"\"\"\\n5tkinter==8.6\\n6Pillow==8.3.1\\n7\"\"\"\\n8‘‘‘\\n9\\n10## Required Other language third-party packages\\n11‘‘‘python\\n12\"\"\"\\n13No third-party packages required in other languages.\\n14\"\"\"\\n15‘‘‘\\n16\\n17## Full API spec\\n18‘‘‘python\\n19\"\"\"\\n20No APIs are used in this project.\\n21\"\"\"\\n22‘‘‘\\n23\\n24## Logic Analysis\\n25‘‘‘python\\n26[\\n27 (\"main.py\", \"Contains the main application loop and initializes all the other components.\"),\\n28 (\"canvas.py\", \"Implements the drawing canvas, responds to mouse events, and uses the selected\\n,→tool and color.\"),\\n29 (\"tools.py\", \"Implements the tool selection interface.\"),\\n30 (\"color_picker.py\", \"Implements the color selection interface.\"),\\n31 (\"file_manager.py\", \"Implements the file saving and opening functionality.\")\\n32]\\n33‘‘‘\\n20', start_char_idx=0, end_char_idx=901, text_template='{metadata_str}\\n\\n{content}', metadata_template='{key}: {value}', metadata_seperator='\\n'),\n",
       " TextNode(id_='b645a3e6-de6e-44c9-96a1-ba682315f3e2', embedding=None, metadata={'page_label': '21', 'file_name': 'metagpt.pdf', 'file_path': 'metagpt.pdf', 'file_type': 'application/pdf', 'file_size': 16911937, 'creation_date': '2024-11-25', 'last_modified_date': '2024-06-24'}, excluded_embed_metadata_keys=['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], excluded_llm_metadata_keys=['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], relationships={<NodeRelationship.SOURCE: '1'>: RelatedNodeInfo(node_id='8737dbb9-6ba9-42e0-8d02-c3be47982544', node_type=<ObjectType.DOCUMENT: '4'>, metadata={'page_label': '21', 'file_name': 'metagpt.pdf', 'file_path': 'metagpt.pdf', 'file_type': 'application/pdf', 'file_size': 16911937, 'creation_date': '2024-11-25', 'last_modified_date': '2024-06-24'}, hash='8c62a3a8316b65a34d049bead288cb4697810534be2779bc7760d8ccc09bbf43'), <NodeRelationship.PREVIOUS: '2'>: RelatedNodeInfo(node_id='f26547d5-cee7-46ed-b0e3-e5bc2b1f758e', node_type=<ObjectType.TEXT: '1'>, metadata={'page_label': '20', 'file_name': 'metagpt.pdf', 'file_path': 'metagpt.pdf', 'file_type': 'application/pdf', 'file_size': 16911937, 'creation_date': '2024-11-25', 'last_modified_date': '2024-06-24'}, hash='cb741a963667fada46987dbb295f76acec2a86b01bb8969c681ca10b25c9c8e0'), <NodeRelationship.NEXT: '3'>: RelatedNodeInfo(node_id='89c7eea4-aa8b-4675-bd8e-b0ff1bdd1ac8', node_type=<ObjectType.TEXT: '1'>, metadata={}, hash='af307144c869fbfb702e653d9e0498e6f3697289b04bfe5e3eb3acd9b14687cb')}, text='Preprint\\nProduct Requirement Document\\n1## Task list\\n2‘‘‘python\\n3[\\n4 \"main.py\",\\n5 \"canvas.py\",\\n6 \"tools.py\",\\n7 \"color_picker.py\",\\n8 \"file_manager.py\"\\n9]\\n10‘‘‘\\n11\\n12## Shared Knowledge\\n13‘‘‘python\\n14\"\"\"\\n15The ’Tkinter’ library is used for creating the GUI of the application. It provides various widgets\\n,→ like frames and canvases which are used in this project.\\n16The ’Pillow’ library is used for handling image files. It is used in the ’file_manager.py’ for\\n,→saving and opening image files.\\n17The ’main.py’ file is the entry point of the application. It initializes all the other components\\n,→and starts the application loop.\\n18\"\"\"\\n19‘‘‘\\n20\\n21## Anything UNCLEAR\\n22The project requirements and implementation approach are clear. However, we need to ensure that\\n,→all team members are familiar with the Tkinter and Pillow libraries. If not, they may\\n,→need some time to learn these libraries before starting the development.\\n21', start_char_idx=0, end_char_idx=928, text_template='{metadata_str}\\n\\n{content}', metadata_template='{key}: {value}', metadata_seperator='\\n'),\n",
       " TextNode(id_='89c7eea4-aa8b-4675-bd8e-b0ff1bdd1ac8', embedding=None, metadata={'page_label': '22', 'file_name': 'metagpt.pdf', 'file_path': 'metagpt.pdf', 'file_type': 'application/pdf', 'file_size': 16911937, 'creation_date': '2024-11-25', 'last_modified_date': '2024-06-24'}, excluded_embed_metadata_keys=['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], excluded_llm_metadata_keys=['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], relationships={<NodeRelationship.SOURCE: '1'>: RelatedNodeInfo(node_id='50e48095-225b-4444-af02-00b392a4eebd', node_type=<ObjectType.DOCUMENT: '4'>, metadata={'page_label': '22', 'file_name': 'metagpt.pdf', 'file_path': 'metagpt.pdf', 'file_type': 'application/pdf', 'file_size': 16911937, 'creation_date': '2024-11-25', 'last_modified_date': '2024-06-24'}, hash='d574f7c5a418991e3bea0c88b06386bde0335c076fe4921c414a051bf4039134'), <NodeRelationship.PREVIOUS: '2'>: RelatedNodeInfo(node_id='b645a3e6-de6e-44c9-96a1-ba682315f3e2', node_type=<ObjectType.TEXT: '1'>, metadata={'page_label': '21', 'file_name': 'metagpt.pdf', 'file_path': 'metagpt.pdf', 'file_type': 'application/pdf', 'file_size': 16911937, 'creation_date': '2024-11-25', 'last_modified_date': '2024-06-24'}, hash='8c62a3a8316b65a34d049bead288cb4697810534be2779bc7760d8ccc09bbf43'), <NodeRelationship.NEXT: '3'>: RelatedNodeInfo(node_id='f87db271-9a67-4538-9291-52540b48b4b2', node_type=<ObjectType.TEXT: '1'>, metadata={}, hash='7d81baaf621f5e4ebfef50052c206207e4eef7c331bb5a9458d795053e121f8e')}, text='Preprint\\nEngineer Given the provided file structure and function definitions, an Engineer agent requires\\nonly fundamental development skills to complete the development tasks. Due to the large number\\nof files, we present only one auto-generated code file here.\\nGenerated code (color picker.py)\\n1import tkinter as tk\\n2from tkinter import colorchooser\\n3class ColorPicker:\\n4 def __init__(self, root: tk.Tk, color: str = ’black’):\\n5 self.frame = tk.Frame(root)\\n6 self.color = color\\n7\\n8 self.color_button = tk.Button(self.frame, text=’Color’, command=self.select_color)\\n9 self.color_button.pack(fill=’x’)\\n10\\n11 def select_color(self):\\n12 color = colorchooser.askcolor()[1]\\n13 if color is not None:\\n14 self.color = color\\n15\\n16 def pack(self, **kwargs):\\n17 self.frame.pack( **kwargs)\\n18\\n19 def get_color(self):\\n20 return self.color\\nQA Engineer Upon receiving the code output from the Engineer, the QA Engineer generates unit\\ntest code and reviews it to identify and fix any bugs, ensuring high-quality software.\\nGenerated Unit tests (test.py)\\n1import unittest\\n2import tkinter as tk\\n3# You might want to move the ColorPicker class to another file, let’s say color_picker.py\\n4# from color_picker import ColorPicker\\n5\\n6class TestColorPicker(unittest.TestCase):\\n7\\n8 def setUp(self):\\n9 self.root = tk.Tk()\\n10 self.cp = ColorPicker(self.root)\\n11\\n12 def tearDown(self):\\n13 # This is important to close the tk window after tests\\n14 self.root.destroy()\\n15\\n16 def test_initial_color(self):\\n17 # Test the default color during initialization\\n18 self.assertEqual(self.cp.get_color(), ’black’)\\n19\\n20 def test_set_and_get_color(self):\\n21 # Test setting a new color and retrieving it\\n22 new_color = ’#ffffff’ # white color\\n23 self.cp.color = new_color\\n24 self.assertEqual(self.cp.get_color(), new_color)\\n25\\n26\\n27if __name__ == ’__main__’:\\n28 unittest.main()\\nOutput Ultimately, as shown in Figure 10, MetaGPT generates a functional application named\\n“Drawing App”.\\n22', start_char_idx=0, end_char_idx=1943, text_template='{metadata_str}\\n\\n{content}', metadata_template='{key}: {value}', metadata_seperator='\\n'),\n",
       " TextNode(id_='f87db271-9a67-4538-9291-52540b48b4b2', embedding=None, metadata={'page_label': '23', 'file_name': 'metagpt.pdf', 'file_path': 'metagpt.pdf', 'file_type': 'application/pdf', 'file_size': 16911937, 'creation_date': '2024-11-25', 'last_modified_date': '2024-06-24'}, excluded_embed_metadata_keys=['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], excluded_llm_metadata_keys=['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], relationships={<NodeRelationship.SOURCE: '1'>: RelatedNodeInfo(node_id='167e851b-114e-4065-b305-04fc68e713a5', node_type=<ObjectType.DOCUMENT: '4'>, metadata={'page_label': '23', 'file_name': 'metagpt.pdf', 'file_path': 'metagpt.pdf', 'file_type': 'application/pdf', 'file_size': 16911937, 'creation_date': '2024-11-25', 'last_modified_date': '2024-06-24'}, hash='bfb971ccd57b036adc3aff6bb6e320d3f52ebff10cb68671adf955c87ef29070'), <NodeRelationship.PREVIOUS: '2'>: RelatedNodeInfo(node_id='89c7eea4-aa8b-4675-bd8e-b0ff1bdd1ac8', node_type=<ObjectType.TEXT: '1'>, metadata={'page_label': '22', 'file_name': 'metagpt.pdf', 'file_path': 'metagpt.pdf', 'file_type': 'application/pdf', 'file_size': 16911937, 'creation_date': '2024-11-25', 'last_modified_date': '2024-06-24'}, hash='d574f7c5a418991e3bea0c88b06386bde0335c076fe4921c414a051bf4039134'), <NodeRelationship.NEXT: '3'>: RelatedNodeInfo(node_id='88a1b404-7c77-46a2-a4dc-62c40cd2b140', node_type=<ObjectType.TEXT: '1'>, metadata={}, hash='16bcbce63ff3d327b476a870791e0fd7878bdf8b5de69c415e2603798d351109')}, text='Preprint\\nFigure 10: The “Drawing App” generated by MetaGPT.\\nC E XPERIMENTS\\nC.1 D ETAILS OF THE SOFTWARE DEVDATASET\\nThe SoftwareDev dataset includes 70 diverse software development tasks. Table 8 displays the\\nnames and detailed prompts of 11 tasks within the dataset. Note that the first seven tasks listed are\\nused in the main experiments of this paper.\\nC.2 A DDITIONAL RESULTS\\nQuantitative results of MetaGPT As shown in Table 4, MetaGPT achieves an average score\\nof 3.9, surpassing ChatDev’s score of 2.1 Zhao et al. (2023), which is based on the Chat chain.\\nCompare the scores of general intelligent algorithms, including AutoGPT Torantulino et al. (2023),\\nwhich all score 1.0, failing to generate executable code. We observe that the generated code is often\\nshort, lacks comprehensive logic, and tends to fail to handle cross-file dependencies correctly.\\nWhile models such as AutoGPT (Torantulino et al., 2023), Langchain (Chase, 2022), and Agent-\\nVerse (Chen et al., 2023) display robust general problem-solving capabilities, they lack an essential\\nelement for developing complex systems: systematically deconstructing requirements. Conversely,\\nMetaGPT simplifies the process of transforming abstract requirements into detailed class and func-\\ntion designs through a specialized division of labor and SOPs workflow. When compared to Chat-\\nDev (Zhao et al., 2023), MetaGPT’s structured messaging and feedback mechanisms not only reduce\\nloss of communication information but also improve the execution of code.\\nQuantitative results of MetaGPT w/o executable feedback Table 9 presents the performance of\\nMetaGPT with GPT-4 32K on 11 tasks within the SoftwareDev dataset. It also shows the average\\nperformance across all 70 tasks (in the last line). Note that the version of MetaGPT used here is the\\nbasic version without the executable feedback mechanism.\\nQuantitative results of MetaGPT with different LLMs To verify the performance of MetaGPT\\non different LLMs, we randomly selected 5 SoftwareDev tasks and conducted experiments using\\nGPT-3.5 and Deepseek Coder 33B5as backends. As shown in Table 5, the results indicate that\\nalthough MetaGPT can complete tasks with these LLMs, using GPT-4 as the backend yields superior\\nperformance.\\n5https://deepseekcoder.github.io\\n23', start_char_idx=0, end_char_idx=2274, text_template='{metadata_str}\\n\\n{content}', metadata_template='{key}: {value}', metadata_seperator='\\n'),\n",
       " TextNode(id_='88a1b404-7c77-46a2-a4dc-62c40cd2b140', embedding=None, metadata={'page_label': '24', 'file_name': 'metagpt.pdf', 'file_path': 'metagpt.pdf', 'file_type': 'application/pdf', 'file_size': 16911937, 'creation_date': '2024-11-25', 'last_modified_date': '2024-06-24'}, excluded_embed_metadata_keys=['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], excluded_llm_metadata_keys=['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], relationships={<NodeRelationship.SOURCE: '1'>: RelatedNodeInfo(node_id='41f18b7a-2bcb-4528-a917-48bf4b72f78b', node_type=<ObjectType.DOCUMENT: '4'>, metadata={'page_label': '24', 'file_name': 'metagpt.pdf', 'file_path': 'metagpt.pdf', 'file_type': 'application/pdf', 'file_size': 16911937, 'creation_date': '2024-11-25', 'last_modified_date': '2024-06-24'}, hash='524a7a4bc0c269f7deaad984b9e3fb778f2094fda4e76b3e4d6dddd95a6f6144'), <NodeRelationship.PREVIOUS: '2'>: RelatedNodeInfo(node_id='f87db271-9a67-4538-9291-52540b48b4b2', node_type=<ObjectType.TEXT: '1'>, metadata={'page_label': '23', 'file_name': 'metagpt.pdf', 'file_path': 'metagpt.pdf', 'file_type': 'application/pdf', 'file_size': 16911937, 'creation_date': '2024-11-25', 'last_modified_date': '2024-06-24'}, hash='bfb971ccd57b036adc3aff6bb6e320d3f52ebff10cb68671adf955c87ef29070'), <NodeRelationship.NEXT: '3'>: RelatedNodeInfo(node_id='8d641f7b-f8a7-4900-ba21-924ba1a726d2', node_type=<ObjectType.TEXT: '1'>, metadata={}, hash='a6c3e912b757d23931322a4c532361509f78d5ba4f22fc0aa71d985b7ad5b85d')}, text='Preprint\\nTable 4: Executability comparison. The executability scores are on a grading system ranging from\\n’1’ to ’4’. A score of ’1’ signifies complete failure, ’2’ denotes executable code, ’3’ represents\\nlargely satisfying expected workflow, and ’4’ indicates a perfect match with expectations.\\nTask AutoGPT LangChain AgentVerse ChatDev MetaGPT\\nFlappy bird 1 1 1 2 3\\nTank battle game 1 1 1 2 4\\n2048 game 1 1 1 1 4\\nSnake game 1 1 1 3 4\\nBrick breaker game 1 1 1 1 4\\nExcel data process 1 1 1 4 4\\nCRUD manage 1 1 1 2 4\\nAverage score 1.0 1.0 1.0 2.1 3.9\\nTable 5: Performance of MetaGPT on SoftwareDev using different LLMs as agent backends.\\nModel Open source Time(/s) # Lines Executability Revisions\\nMetaGPT (w/ GPT-3.5) % 75.18 161.6 2.8 2.4\\nMetaGPT (w/ GPT-4) % 552.94 178.2 3.8 1.2\\nMetaGPT (w/ Deepseek Coder 33B) \" 1186.20 120.2 1.4 2.6\\nImpact of Instruction Levels (High-level v.s.Detailed Instructions) Does the variation in the\\nlevel of initial input from humans significantly influence performance outcomes? For examples:\\n1.High-level prompt : Create a brick breaker game.\\n2.Detailed prompt : Creating a brick breaker game. In a brick breaker game, the player\\ntypically controls a paddle at the bottom of the screen to bounce a ball towards a wall of\\nbricks. The goal is to break all the bricks by hitting them with the ball.\\nAdditional experiments were conducted to investigate this aspect: we selected 5 tasks from Soft-\\nwareDev, and constructed detailed prompts for them. Here are the experimental results:\\nTable 6: Impact of Instruction Levels. The executability is scored on a grading system ranging\\nfrom ‘1’ to ‘4’. A score of ‘1’ signifies complete failure, ‘2’ denotes runnable code, ‘3’ represents\\nlargely expected workflow, and ‘4’ indicates a perfect match to expectations.\\nModel # Word Time(/s) Token usage # Lines Executability Productivity Reversions\\nHigh-level 13.2 552.9 28384.2 178.2 3.8 163.8 1.2\\nDetailed 42.2 567.8 29657.0 257.0 4.0 118.0 1.6\\nWe observe that: detailed prompts lead to better software projects with lower productivity ratios\\nbecause of clearer requirements and functions, while simple inputs can still generate good enough\\nsoftware using MetaGPT with an executability rating of 3.8, which is comparable to the detailed\\nprompt scenario. (Note that, Productivity = Token usage / Total Code Lines. The lower this ratio,\\nthe better.)\\nThe performance of GPT variants in HumanEval benchmark We use the GPT-4’s 67% Hu-\\nmanEval score (OpenAI, 2023) as our baseline, acknowledging its acceptance in the HumanEval\\nbenchmark. We further extend to experiments(five times) with GPT-4 (gpt-4-0613) and GPT-3.5-\\nTurbo (gpt-3.5-turbo-0613) under various conditions to assess performance. (A)We directly called\\nthe OpenAI API with the prompt in HumanEval. (B)We called the OpenAI API and parsed the\\ncode with regex in the response. (C)We added an additional system prompt, then called the OpenAI\\nAPI. The prompt is ”You are an AI that only responds with Python code, NOT ENGLISH. You will\\n24', start_char_idx=0, end_char_idx=3014, text_template='{metadata_str}\\n\\n{content}', metadata_template='{key}: {value}', metadata_seperator='\\n'),\n",
       " TextNode(id_='8d641f7b-f8a7-4900-ba21-924ba1a726d2', embedding=None, metadata={'page_label': '25', 'file_name': 'metagpt.pdf', 'file_path': 'metagpt.pdf', 'file_type': 'application/pdf', 'file_size': 16911937, 'creation_date': '2024-11-25', 'last_modified_date': '2024-06-24'}, excluded_embed_metadata_keys=['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], excluded_llm_metadata_keys=['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], relationships={<NodeRelationship.SOURCE: '1'>: RelatedNodeInfo(node_id='62d39dfc-1651-4aea-96a1-9d04bd094074', node_type=<ObjectType.DOCUMENT: '4'>, metadata={'page_label': '25', 'file_name': 'metagpt.pdf', 'file_path': 'metagpt.pdf', 'file_type': 'application/pdf', 'file_size': 16911937, 'creation_date': '2024-11-25', 'last_modified_date': '2024-06-24'}, hash='eec96a5fe356a9267b7bf567e7e64c0a84bdca4ed621d2c087e08adbf7af2744'), <NodeRelationship.PREVIOUS: '2'>: RelatedNodeInfo(node_id='88a1b404-7c77-46a2-a4dc-62c40cd2b140', node_type=<ObjectType.TEXT: '1'>, metadata={'page_label': '24', 'file_name': 'metagpt.pdf', 'file_path': 'metagpt.pdf', 'file_type': 'application/pdf', 'file_size': 16911937, 'creation_date': '2024-11-25', 'last_modified_date': '2024-06-24'}, hash='524a7a4bc0c269f7deaad984b9e3fb778f2094fda4e76b3e4d6dddd95a6f6144'), <NodeRelationship.NEXT: '3'>: RelatedNodeInfo(node_id='1b9bbafe-b670-41ec-acd0-2c8e6658c2b8', node_type=<ObjectType.TEXT: '1'>, metadata={}, hash='5e8eb92cf4b1fcca32aa4d2071b78184d60d98c1782e321278210922bbac1cee')}, text='Preprint\\nbe given a function signature and its docstring by the user. Write your full implementation (restate\\nthe function signature).” As shown in Table 7, GPT-4 is more sensitive to prompt, code parser, and\\npost-processing results on the HumanEval data set. It is difficult for GPT-3.5-Turbo to return the\\ncorrect completion code without prompt words.\\nTable 7: Performance of GPT models on HumanEval. Experiments were conducted five times\\nusing gpt-4-0613 and gpt-3.5-turbo-0613 with different settings.\\nSettings Model 1 2 3 4 5 Avg. Std.\\nA gpt-4-0613 0.732 0.707 0.732 0.713 0.738 0.724 0.013\\nA gpt-3.5-turbo-0613 0.360 0.366 0.360 0.348 0.354 0.357 0.007\\nB gpt-4-0613 0.787 0.811 0.817 0.829 0.817 0.812 0.016\\nB gpt-3.5-turbo-0613 0.348 0.354 0.348 0.335 0.348 0.346 0.007\\nC gpt-4-0613 0.805 0.805 0.817 0.793 0.780 0.800 0.014\\nC gpt-3.5-turbo-0613 0.585 0.567 0.573 0.579 0.579 0.577 0.007\\nQualitative results Figure 11 and Figure 12 illustrate the outcomes of the Architect agent’s ef-\\nforts to design a complex recommender system. These figures showcase the comprehensive system\\ninterface design and program call flow. The latter is essential for creating a sophisticated automated\\nsystem. It is crucial to emphasize the importance of this division of labor in developing an automated\\nsoftware framework.\\nD L IMITATION AND ETHICS CONCERNS\\nD.1 L IMITATION\\nSystem side At present, our system cannot fully cater to specific scenarios, such as UI and front-\\nend, as we have yet to incorporate such agents and multimodal tools. Furthermore, despite gen-\\nerating the most amount of code among comparable frameworks, it remains challenging to fulfill\\nreal-world applications’ diverse and complex requirements.\\nHuman user side A key challenge for users is to interrupt the running process of each agent, or\\nset the starting running point (checkpoint) for each agent.\\nD.2 E THICS CONCERNS\\nUnemployment and Skill Obsolescence MetaGPT enables more people to program in natural\\nlanguages, thereby making it easier for engineers to get started. Over the years, programming\\nlanguages have evolved from punched cards to assembly, C, Java, Python, and now natural lan-\\nguage. As a result, humans have become more proficient at programming, increasing the demand\\nfor programming-related positions. Furthermore, programming with natural language may offer a\\nsignificantly easier learning curve, making programming more accessible to a broader audience.\\nTransparency and Accountability MetaGPT is an open-source framework that facilitates inter-\\nactive communication between multiple agents through natural language. Humans can initiate, ob-\\nserve, and stop running with the highest level of control. It provides real-time interpretation and op-\\neration of the natural language, displayed on the screen and logs, ensuring transparency. MetaGPT\\nenhances “natural language programming” capabilities, and human engineers are the users and re-\\nsponsible for the outcomes.\\nPrivacy and Data Security MetaGPT operates locally, ensuring user data privacy and security. It\\ndoes not collect user data. For interactions with third-party LLMs, such as those by OpenAI, users\\nare encouraged to refer to the respective privacy policies (e.g., OpenAI Privacy Policy). However,\\nwe provide the option of open-source LLMs as backends.\\n25', start_char_idx=0, end_char_idx=3307, text_template='{metadata_str}\\n\\n{content}', metadata_template='{key}: {value}', metadata_seperator='\\n'),\n",
       " TextNode(id_='1b9bbafe-b670-41ec-acd0-2c8e6658c2b8', embedding=None, metadata={'page_label': '26', 'file_name': 'metagpt.pdf', 'file_path': 'metagpt.pdf', 'file_type': 'application/pdf', 'file_size': 16911937, 'creation_date': '2024-11-25', 'last_modified_date': '2024-06-24'}, excluded_embed_metadata_keys=['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], excluded_llm_metadata_keys=['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], relationships={<NodeRelationship.SOURCE: '1'>: RelatedNodeInfo(node_id='e85ff559-0877-4a8a-9d65-68aa0d5364bb', node_type=<ObjectType.DOCUMENT: '4'>, metadata={'page_label': '26', 'file_name': 'metagpt.pdf', 'file_path': 'metagpt.pdf', 'file_type': 'application/pdf', 'file_size': 16911937, 'creation_date': '2024-11-25', 'last_modified_date': '2024-06-24'}, hash='a71da697e9fe843fe1c56c293229a7715924df1ed82ab4c423eb6974034edf7e'), <NodeRelationship.PREVIOUS: '2'>: RelatedNodeInfo(node_id='8d641f7b-f8a7-4900-ba21-924ba1a726d2', node_type=<ObjectType.TEXT: '1'>, metadata={'page_label': '25', 'file_name': 'metagpt.pdf', 'file_path': 'metagpt.pdf', 'file_type': 'application/pdf', 'file_size': 16911937, 'creation_date': '2024-11-25', 'last_modified_date': '2024-06-24'}, hash='eec96a5fe356a9267b7bf567e7e64c0a84bdca4ed621d2c087e08adbf7af2744'), <NodeRelationship.NEXT: '3'>: RelatedNodeInfo(node_id='946bd8a1-c658-49b1-a4c6-7220d719a27a', node_type=<ObjectType.TEXT: '1'>, metadata={}, hash='8faba8f2bccafbb82a85f9a2b2ed0909777406ab88786d6b0504a56793181f1d')}, text='Preprint\\nFigure 11: The system interface design for “recommendation engine development” is generated by\\nthearchitect agent ( zoom in for a better view ).\\nE M ORE DISCUSSIONS\\nE.1 D EEP-SEATED CHALLENGES\\nMetaGPT also alleviates or solves these challenges with its unique designs:\\nUse Context Efficiently Two sub-challenges are present. First, unfolding short natural language\\ndescriptions accurately to eliminate ambiguity. Second, maintaining information validity in lengthy\\ncontexts, enables LLMs to concentrate on relevant data without distraction.\\nReduce Hallucinations Using LLMs to generate entire software programs faces code halluci-\\nnation problems—-including incomplete implementation of functions, missing dependencies, and\\npotential undiscovered bugs, which may be more serious. LLMs often struggle with software gen-\\neration due to vague task definitions. Focusing on granular tasks like requirement analysis and\\npackage selection offers guided thinking, which LLMs lack in broad task solving.\\nE.2 I NFORMATION OVERLOAD\\nIn MetaGPT, we use a global message pool and a subscription mechanism to address “information\\noverload,” which refers to the problem of receiving excessive or irrelevant information. This issue\\nis dependent on specific applications. MetaGPT employs a message pool to streamline communi-\\ncation, ensuring efficiency. Additionally, a subscription mechanism filters out irrelevant contexts,\\nenhancing the relevance and utility of the information. This design is particularly crucial in soft-\\n26', start_char_idx=0, end_char_idx=1522, text_template='{metadata_str}\\n\\n{content}', metadata_template='{key}: {value}', metadata_seperator='\\n'),\n",
       " TextNode(id_='946bd8a1-c658-49b1-a4c6-7220d719a27a', embedding=None, metadata={'page_label': '27', 'file_name': 'metagpt.pdf', 'file_path': 'metagpt.pdf', 'file_type': 'application/pdf', 'file_size': 16911937, 'creation_date': '2024-11-25', 'last_modified_date': '2024-06-24'}, excluded_embed_metadata_keys=['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], excluded_llm_metadata_keys=['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], relationships={<NodeRelationship.SOURCE: '1'>: RelatedNodeInfo(node_id='7d26b232-1654-4585-90f7-1813b0773c25', node_type=<ObjectType.DOCUMENT: '4'>, metadata={'page_label': '27', 'file_name': 'metagpt.pdf', 'file_path': 'metagpt.pdf', 'file_type': 'application/pdf', 'file_size': 16911937, 'creation_date': '2024-11-25', 'last_modified_date': '2024-06-24'}, hash='91d7b000c190e629ea104bdd13ecdbbf050bfe4c4a44c4533ae35694b509fa6b'), <NodeRelationship.PREVIOUS: '2'>: RelatedNodeInfo(node_id='1b9bbafe-b670-41ec-acd0-2c8e6658c2b8', node_type=<ObjectType.TEXT: '1'>, metadata={'page_label': '26', 'file_name': 'metagpt.pdf', 'file_path': 'metagpt.pdf', 'file_type': 'application/pdf', 'file_size': 16911937, 'creation_date': '2024-11-25', 'last_modified_date': '2024-06-24'}, hash='a71da697e9fe843fe1c56c293229a7715924df1ed82ab4c423eb6974034edf7e'), <NodeRelationship.NEXT: '3'>: RelatedNodeInfo(node_id='251346fc-79ed-41d9-af98-3b05288f3e5d', node_type=<ObjectType.TEXT: '1'>, metadata={}, hash='7344d1b748830bb6d0143c67ae82b2d106bcae6587adfdc46879af49912e8fd1')}, text='Preprint\\nFigure 12: The program call flow for “recommendation engine development” generated by the\\narchitect agent ( zoom in for a better view ).\\nware design scenarios and standard operating procedures (SOPs) where effective communication is\\nessential.\\n27', start_char_idx=0, end_char_idx=255, text_template='{metadata_str}\\n\\n{content}', metadata_template='{key}: {value}', metadata_seperator='\\n'),\n",
       " TextNode(id_='251346fc-79ed-41d9-af98-3b05288f3e5d', embedding=None, metadata={'page_label': '28', 'file_name': 'metagpt.pdf', 'file_path': 'metagpt.pdf', 'file_type': 'application/pdf', 'file_size': 16911937, 'creation_date': '2024-11-25', 'last_modified_date': '2024-06-24'}, excluded_embed_metadata_keys=['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], excluded_llm_metadata_keys=['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], relationships={<NodeRelationship.SOURCE: '1'>: RelatedNodeInfo(node_id='b4faedc8-4015-4005-a2c1-2de15da2823a', node_type=<ObjectType.DOCUMENT: '4'>, metadata={'page_label': '28', 'file_name': 'metagpt.pdf', 'file_path': 'metagpt.pdf', 'file_type': 'application/pdf', 'file_size': 16911937, 'creation_date': '2024-11-25', 'last_modified_date': '2024-06-24'}, hash='b889a05eeda7027dd0e53213e519b4627cd527a8bc994e9f56565f850d362057'), <NodeRelationship.PREVIOUS: '2'>: RelatedNodeInfo(node_id='946bd8a1-c658-49b1-a4c6-7220d719a27a', node_type=<ObjectType.TEXT: '1'>, metadata={'page_label': '27', 'file_name': 'metagpt.pdf', 'file_path': 'metagpt.pdf', 'file_type': 'application/pdf', 'file_size': 16911937, 'creation_date': '2024-11-25', 'last_modified_date': '2024-06-24'}, hash='91d7b000c190e629ea104bdd13ecdbbf050bfe4c4a44c4533ae35694b509fa6b'), <NodeRelationship.NEXT: '3'>: RelatedNodeInfo(node_id='8b0b0ed7-1dd9-4fdc-9b0c-821ae3945a6b', node_type=<ObjectType.TEXT: '1'>, metadata={}, hash='5b5b2b136051432e3398f319fc7982a2d4ce91f632471ef7f31fb872a9876019')}, text='Preprint\\nTable 8: Examples of SoftwareDev dataset.\\nTask ID Task Prompt\\n0 Snake game Create a snake game.\\n1 Brick breaker game Create a brick breaker game.\\n2 2048 game Create a 2048 game for the web.\\n3 Flappy bird game Write p5.js code for Flappy Bird where you control a yellow bird continu-\\nously flying between a series of green pipes. The bird flaps every time you\\nleft click the mouse. If it falls to the ground or hits a pipe, you lose. This\\ngame goes on indefinitely until you lose; you get points the further you go.\\n4 Tank battle game Create a tank battle game.\\n5 Excel data process Write an excel data processing program based on streamlit and pandas. The\\nscreen first shows an excel file upload button. After the excel file is uploaded,\\nuse pandas to display its data content. The program is required to be concise,\\neasy to maintain, and not over-designed. It uses streamlit to process web\\nscreen displays, and pandas is sufficient to process excel reading and display.\\nPlease make sure others can execute directly without introducing additional\\npackages.\\n6 CRUD manage Write a management program based on the crud addition, deletion, modifi-\\ncation and query processing of the customer business entity. The customer\\nneeds to save this information: name, birthday, age, sex, and phone. The data\\nis stored in client.db, and there is a judgement whether the customer table ex-\\nists. If it doesn’t, it needs to be created first. Querying is done by name; same\\nfor deleting. The program is required to be concise, easy to maintain, and not\\nover-designed. The screen is realized through streamlit and sqlite—no need\\nto introduce other additional packages.\\n7 Music transcriber Develop a program to transcribe sheet music into a digital format; provid-\\ning error-free transcribed symbolized sheet music intelligence from audio\\nthrough signal processing involving pitch and time slicing then training a\\nneural net to run Onset Detected CWT transforming scalograms to chroma-\\ngrams decoded with Recursive Neural Network focused networks.\\n8 Custom press releases Create custom press releases; develop a Python script that extracts rele-\\nvant information about company news from external sources, such as social\\nmedia; extract update interval database for recent changes. The program\\nshould create press releases with customizable options and export writings\\nto PDFs, NYTimes API JSONs, media format styled with interlink internal\\nfixed character-length metadata.\\n9 Gomoku game Implement a Gomoku game using Python, incorporating an AI opponent\\nwith varying difficulty levels.\\n10 Weather dashboard Create a Python program to develop an interactive weather dashboard.\\n28', start_char_idx=0, end_char_idx=2668, text_template='{metadata_str}\\n\\n{content}', metadata_template='{key}: {value}', metadata_seperator='\\n'),\n",
       " TextNode(id_='8b0b0ed7-1dd9-4fdc-9b0c-821ae3945a6b', embedding=None, metadata={'page_label': '29', 'file_name': 'metagpt.pdf', 'file_path': 'metagpt.pdf', 'file_type': 'application/pdf', 'file_size': 16911937, 'creation_date': '2024-11-25', 'last_modified_date': '2024-06-24'}, excluded_embed_metadata_keys=['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], excluded_llm_metadata_keys=['file_name', 'file_type', 'file_size', 'creation_date', 'last_modified_date', 'last_accessed_date'], relationships={<NodeRelationship.SOURCE: '1'>: RelatedNodeInfo(node_id='17bd35b2-3324-4b03-b1b8-3d7a2aac2361', node_type=<ObjectType.DOCUMENT: '4'>, metadata={'page_label': '29', 'file_name': 'metagpt.pdf', 'file_path': 'metagpt.pdf', 'file_type': 'application/pdf', 'file_size': 16911937, 'creation_date': '2024-11-25', 'last_modified_date': '2024-06-24'}, hash='b6e4fef1f855f205bc2941cc9ba967889df8d25912481aa6582ef327c1384a9f'), <NodeRelationship.PREVIOUS: '2'>: RelatedNodeInfo(node_id='251346fc-79ed-41d9-af98-3b05288f3e5d', node_type=<ObjectType.TEXT: '1'>, metadata={'page_label': '28', 'file_name': 'metagpt.pdf', 'file_path': 'metagpt.pdf', 'file_type': 'application/pdf', 'file_size': 16911937, 'creation_date': '2024-11-25', 'last_modified_date': '2024-06-24'}, hash='b889a05eeda7027dd0e53213e519b4627cd527a8bc994e9f56565f850d362057')}, text='Preprint\\nTable 9: Additional results of pure MetaGPT w/o feedback on SoftwareDev. Averages (Avg.) of 70 tasks are calculated and 10 randomly selected tasks are\\nincluded. ‘#’ denotes ‘The number of’, while ‘ID’ is ‘Task ID’.\\nID Code statistics Doc statistics Cost statistics Cost of revision Code executability\\n#code files #lines of code #lines per code file #doc files #lines of doc #lines per doc file #prompt tokens #completion tokens time costs money costs\\n0 5.00 196.00 39.20 3.00 210.00 70.00 24087.00 6157.00 582.04 $ 1.09 1. TypeError 4\\n1 6.00 191.00 31.83 3.00 230.00 76.67 32517.00 6238.00 566.30 $ 1.35 1. TypeError 4\\n2 3.00 198.00 66.00 3.00 235.00 78.33 21934.00 6316.00 553.11 $ 1.04 1. lack\\n@app.route(’/’)3\\n3 5.00 164 32.80 3.00 202.00 67.33 22951.00 5312.00 481.34 $ 1.01 1. PNG file miss-\\ning 2. Compile bug\\nfixes2\\n4 6.00 203.00 33.83 3.00 210.00 70.00 30087.00 6567.00 599.58 $ 1.30 1. PNG file\\nmissing 2. Com-\\npile bug fixes 3.\\npygame.surface not\\ninitialize3\\n5 6.00 219.00 36.50 3.00 294.00 96.00 35590.00 7336.00 585.10 $ 1.51 1. dependency er-\\nror 2. ModuleNot-\\nFoundError4\\n6 4.00 73.00 18.25 3.00 261.00 87.00 25673.00 5832.00 398.83 $ 0.90 0 4\\n7 4.00 316.00 79.00 3.00 332.00 110.67 29139.00 7104.00 435.83 $ 0.92 0 4\\n8 5.00 215.00 43.00 3.00 301.00 100.33 29372.00 6499.00 621.73 $ 1.27 1. tensorflow ver-\\nsion error 2. model\\ntraining method not\\nimplement2\\n9 5.00 215.00 43.00 3.00 270.00 90.00 24799.00 5734.00 550.88 $ 1.27 1. dependency er-\\nror 2. URL 403 er-\\nror3\\n10 3.00 93.00 31.00 3.00 254.00 84.67 24109.00 5363.00 438.50 $ 0.92 1. dependency er-\\nror 2. missing main\\nfunc.4\\nAvg. 4.71 191.57 42.98 3.00 240.00 80.00 26626.86 6218.00 516.71 $1.12 0.51 (only consider\\nitem scored 2, 3 or\\n4)3.36\\n29', start_char_idx=0, end_char_idx=1726, text_template='{metadata_str}\\n\\n{content}', metadata_template='{key}: {value}', metadata_seperator='\\n')]"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "nodes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "7d1cb142",
   "metadata": {
    "height": 29
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "34\n",
      "Preprint\n",
      "METAGPT: M ETA PROGRAMMING FOR A\n",
      "MULTI -AGENT COLLABORATIVE FRAMEWORK\n",
      "Sirui Hong1∗, Mingchen Zhuge2∗, Jonathan Chen1, Xiawu Zheng3, Yuheng Cheng4,\n",
      "Ceyao Zhang4,Jinlin Wang1,Zili Wang ,Steven Ka Shing Yau5,Zijuan Lin4,\n",
      "Liyang Zhou6,Chenyu Ran1,Lingfeng Xiao1,7,Chenglin Wu1†,J¨urgen Schmidhuber2,8\n",
      "1DeepWisdom,2AI Initiative, King Abdullah University of Science and Technology,\n",
      "3Xiamen University,4The Chinese University of Hong Kong, Shenzhen,\n",
      "5Nanjing University,6University of Pennsylvania,\n",
      "7University of California, Berkeley,8The Swiss AI Lab IDSIA/USI/SUPSI\n",
      "ABSTRACT\n",
      "Remarkable progress has been made on automated problem solving through so-\n",
      "cieties of agents based on large language models (LLMs). Existing LLM-based\n",
      "multi-agent systems can already solve simple dialogue tasks. Solutions to more\n",
      "complex tasks, however, are complicated through logic inconsistencies due to\n",
      "cascading hallucinations caused by naively chaining LLMs. Here we introduce\n",
      "MetaGPT, an innovative meta-programming framework incorporating efficient\n",
      "human workflows into LLM-based multi-agent collaborations. MetaGPT en-\n",
      "codes Standardized Operating Procedures (SOPs) into prompt sequences for more\n",
      "streamlined workflows, thus allowing agents with human-like domain expertise\n",
      "to verify intermediate results and reduce errors. MetaGPT utilizes an assembly\n",
      "line paradigm to assign diverse roles to various agents, efficiently breaking down\n",
      "complex tasks into subtasks involving many agents working together. On col-\n",
      "laborative software engineering benchmarks, MetaGPT generates more coherent\n",
      "solutions than previous chat-based multi-agent systems. Our project can be found\n",
      "at https://github.com/geekan/MetaGPT.\n",
      "1 I NTRODUCTION\n",
      "Autonomous agents utilizing Large Language Models (LLMs) offer promising opportunities to en-\n",
      "hance and replicate human workflows. In real-world applications, however, existing systems (Park\n",
      "et al., 2023; Zhuge et al., 2023; Cai et al., 2023; Wang et al., 2023c; Li et al., 2023; Du et al., 2023;\n",
      "Liang et al., 2023; Hao et al., 2023) tend to oversimplify the complexities. They struggle to achieve\n",
      "effective, coherent, and accurate problem-solving processes, particularly when there is a need for\n",
      "meaningful collaborative interaction (Chen et al., 2024; Zhang et al., 2023; Dong et al., 2023; Zhou\n",
      "et al., 2023; Qian et al., 2023).\n",
      "Through extensive collaborative practice, humans have developed widely accepted Standardized\n",
      "Operating Procedures (SOPs) across various domains (Belbin, 2012; Manifesto, 2001; DeMarco &\n",
      "Lister, 2013). These SOPs play a critical role in supporting task decomposition and effective coor-\n",
      "dination. Furthermore, SOPs outline the responsibilities of each team member, while establishing\n",
      "standards for intermediate outputs. Well-defined SOPs improve the consistent and accurate exe-\n",
      "cution of tasks that align with defined roles and quality standards (Belbin, 2012; Manifesto, 2001;\n",
      "DeMarco & Lister, 2013; Wooldridge & Jennings, 1998). For instance, in a software company,\n",
      "Product Managers analyze competition and user needs to create Product Requirements Documents\n",
      "(PRDs) using a standardized structure, to guide the developmental process.\n",
      "Inspired by such ideas, we design a promising GPT -based Meta -Programming framework called\n",
      "MetaGPT that significantly benefits from SOPs. Unlike other works (Li et al., 2023; Qian et al.,\n",
      "2023), MetaGPT requires agents to generate structured outputs, such as high-quality requirements\n",
      "∗These authors contributed equally to this work.\n",
      "†Chenglin Wu (alexanderwu@fuzhi.ai) is the corresponding author, affiliated with DeepWisdom.\n",
      "1\n"
     ]
    }
   ],
   "source": [
    "print(len(nodes), nodes[0].text, sep='\\n')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "de0660ee-b231-4351-b158-d8ad023e00b5",
   "metadata": {
    "height": 114,
    "tags": []
   },
   "outputs": [],
   "source": [
    "from llama_index.core import Settings\n",
    "from llama_index.llms.openai import OpenAI\n",
    "from llama_index.embeddings.openai import OpenAIEmbedding\n",
    "\n",
    "Settings.llm = OpenAI(model=\"gpt-3.5-turbo\")\n",
    "Settings.embed_model = OpenAIEmbedding(model=\"text-embedding-ada-002\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "997c7559",
   "metadata": {},
   "source": [
    "## Define Summary Index and Vector Index over the Same Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "73d01b01-bc74-432a-8d92-07b9e86498b0",
   "metadata": {
    "height": 80,
    "tags": []
   },
   "outputs": [],
   "source": [
    "from llama_index.core import SummaryIndex, VectorStoreIndex\n",
    "\n",
    "summary_index = SummaryIndex(nodes)\n",
    "vector_index = VectorStoreIndex(nodes)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f9898d3f",
   "metadata": {},
   "source": [
    "## Define Query Engines and Set Metadata"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "44cd7046-c714-4920-b077-b3ded917862f",
   "metadata": {
    "height": 97,
    "tags": []
   },
   "outputs": [],
   "source": [
    "summary_query_engine = summary_index.as_query_engine(\n",
    "    response_mode=\"tree_summarize\",\n",
    "    use_async=True,\n",
    ")\n",
    "vector_query_engine = vector_index.as_query_engine()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "6a1d6d75-247e-426a-8ef4-b49225c24796",
   "metadata": {
    "height": 284,
    "tags": []
   },
   "outputs": [],
   "source": [
    "from llama_index.core.tools import QueryEngineTool\n",
    "\n",
    "\n",
    "summary_tool = QueryEngineTool.from_defaults(\n",
    "    query_engine=summary_query_engine,\n",
    "    description=(\n",
    "        \"Useful for summarization questions related to MetaGPT\"\n",
    "    ),\n",
    ")\n",
    "\n",
    "vector_tool = QueryEngineTool.from_defaults(\n",
    "    query_engine=vector_query_engine,\n",
    "    description=(\n",
    "        \"Useful for retrieving specific context from the MetaGPT paper.\"\n",
    "    ),\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "98d2c152",
   "metadata": {},
   "source": [
    "## Define Router Query Engine"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "00734d7c-638a-4d63-ab1f-7f5a92a65119",
   "metadata": {
    "height": 216,
    "tags": []
   },
   "outputs": [],
   "source": [
    "from llama_index.core.query_engine.router_query_engine import RouterQueryEngine\n",
    "from llama_index.core.selectors import LLMSingleSelector\n",
    "\n",
    "\n",
    "query_engine = RouterQueryEngine(\n",
    "    selector=LLMSingleSelector.from_defaults(),\n",
    "    query_engine_tools=[\n",
    "        summary_tool,\n",
    "        vector_tool,\n",
    "    ],\n",
    "    verbose=True\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "fe3f0a76-68a8-444d-867f-d084bb3ff112",
   "metadata": {
    "height": 46,
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1;3;38;5;200mSelecting query engine 0: Useful for summarization questions related to MetaGPT.\n",
      "\u001b[0mThe document introduces MetaGPT, a meta-programming framework that enhances the problem-solving capabilities of multi-agent systems based on Large Language Models (LLMs). It incorporates Standardized Operating Procedures (SOPs) to streamline workflows, assign roles efficiently, and improve code generation quality. MetaGPT utilizes a structured communication protocol and an executable feedback mechanism to enhance collaboration among agents. The framework has shown state-of-the-art performance in evaluations and offers a promising approach for developing LLM-based multi-agent systems with human-like expertise and robustness. The document discusses the software development process using MetaGPT, emphasizing the importance of SOPs, workflow management, and communication protocols in achieving high-quality software outcomes. It also compares the performance of MetaGPT with other AI models, highlighting its effectiveness in generating executable code and the significance of detailed prompts in software project success. Additionally, the document addresses ethical concerns related to the system, emphasizing transparency, accountability, privacy, and data security in the development and deployment of MetaGPT.\n"
     ]
    }
   ],
   "source": [
    "response = query_engine.query(\"What is the summary of the document?\")\n",
    "print(str(response))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "d3fedea0-f2a9-46bb-8aaf-287df65b8fff",
   "metadata": {
    "height": 29,
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "34\n"
     ]
    }
   ],
   "source": [
    "print(len(response.source_nodes))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "af8c31b3-8e22-4ad9-9825-b8de21bd03c0",
   "metadata": {
    "height": 80,
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1;3;38;5;200mSelecting query engine 1: This choice is more relevant as it focuses on retrieving specific context from the MetaGPT paper, which may contain information on how agents share information with other agents..\n",
      "\u001b[0mAgents share information with other agents by utilizing a shared message pool where they can publish structured messages and also subscribe to relevant messages based on their profiles. This shared message pool allows all agents to exchange messages directly, enabling them to access messages from other entities transparently. Additionally, agents can select information to follow based on their role profiles, ensuring that they receive only task-related information and avoid distractions through irrelevant details.\n"
     ]
    }
   ],
   "source": [
    "response = query_engine.query(\n",
    "    \"How do agents share information with other agents?\"\n",
    ")\n",
    "print(str(response))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aed060ee",
   "metadata": {},
   "source": [
    "## Let's put everything together"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "480b2f45",
   "metadata": {
    "height": 760
   },
   "outputs": [],
   "source": [
    "def get_router_query_engine(file_path: str, llm = None, embed_model = None):\n",
    "    \"\"\"Get router query engine.\"\"\"\n",
    "    llm = llm or OpenAI(model=\"gpt-3.5-turbo\")\n",
    "    embed_model = embed_model or OpenAIEmbedding(model=\"text-embedding-ada-002\")\n",
    "    \n",
    "    # load documents\n",
    "    documents = SimpleDirectoryReader(input_files=[file_path]).load_data()\n",
    "    \n",
    "    splitter = SentenceSplitter(chunk_size=1024)\n",
    "    nodes = splitter.get_nodes_from_documents(documents)\n",
    "    \n",
    "    summary_index = SummaryIndex(nodes)\n",
    "    vector_index = VectorStoreIndex(nodes, embed_model=embed_model)\n",
    "    \n",
    "    summary_query_engine = summary_index.as_query_engine(\n",
    "        response_mode=\"tree_summarize\",\n",
    "        use_async=True,\n",
    "        llm=llm\n",
    "    )\n",
    "    vector_query_engine = vector_index.as_query_engine(llm=llm)\n",
    "    \n",
    "    summary_tool = QueryEngineTool.from_defaults(\n",
    "        query_engine=summary_query_engine,\n",
    "        description=(\n",
    "            \"Useful for summarization questions related to MetaGPT\"\n",
    "        ),\n",
    "    )\n",
    "    \n",
    "    vector_tool = QueryEngineTool.from_defaults(\n",
    "        query_engine=vector_query_engine,\n",
    "        description=(\n",
    "            \"Useful for retrieving specific context from the MetaGPT paper.\"\n",
    "        ),\n",
    "    )\n",
    "    \n",
    "    query_engine = RouterQueryEngine(\n",
    "        selector=LLMSingleSelector.from_defaults(),\n",
    "        query_engine_tools=[\n",
    "            summary_tool,\n",
    "            vector_tool,\n",
    "        ],\n",
    "        verbose=True\n",
    "    )\n",
    "    return query_engine"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "d8f92e0b-1c54-489b-b8dd-41ebaafb380a",
   "metadata": {
    "height": 63
   },
   "outputs": [],
   "source": [
    "# from utils import get_router_query_engine\n",
    "\n",
    "query_engine = get_router_query_engine(\"metagpt.pdf\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "ec1a43f3-77dc-472a-8adc-56551c00a0ff",
   "metadata": {
    "height": 46,
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1;3;38;5;200mSelecting query engine 1: Ablation study results are specific context from the MetaGPT paper, making choice 2 the most relevant..\n",
      "\u001b[0mThe ablation study results provide insights into the impact of removing certain components or features from a system or model. This analysis helps in understanding the contribution and significance of individual elements towards the overall performance or functionality of the system.\n"
     ]
    }
   ],
   "source": [
    "response = query_engine.query(\"Tell me about the ablation study results?\")\n",
    "print(str(response))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
